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PREFACE 


In the fail of 1989 we were approached by John Hollerbach, a member of the Engineering 
Conferences Advisory Committee of the Engineering Foundation, and asked if we would consider 
organizing a meeting around the general theme of teleoperation. After a number of discussions with 
a variety of people at MIT and elsewhere, we decided that it was time to bring together in an 
informal setting two research communities that are working on distinct but closely related types of 
systems: teleoperators and virtual environments. 

Accordingly, we invited the best people available in these two emerging fields. We were 
extremely pleased that, with relatively few exceptions, all people contacted were enthusiastic about 
attending the proposed meeting. 

We decided to have three kinds of sessions: invited-paper sessions, panel discussions, and poster 
sessions. The invited papers were divided into papers providing general background on sensorimotor 
performance and human-machine interfaces, papers on current problems and progress in 
teleoperation, and the same in virtual environments. Also, the panel discussions were subdivided into 
three problem areas: (1) telepresence and sensorimotor adaptation; (2) performance measurement 
and evaluation methods; and (3) design principles and predictive models. A mini seminar on 
teleoperators with micro slave robots was also included. 

In order not to burden the people making presentations, we did not require preparation of written 
papers. Accordingly, this report in no sense constitutes a proceedings. It is, rather, a collection of 
conference-related fragments, highly non-uniform and totally unedited, that we hope will give 
people, particularly those who could not attend, a glimpse of the potential of these fields and how 
they interplay. We have included titles of presentations even when no written material was made 
available (either because no written material was submitted or because the presentation was 
withdrawn). Interested readers should contact the authors directly to obtain further information on 
these items. It is expected that a book on this topic, nourished by this conference, will be published 
in the not-too-distant future. 

We are grateful to Dr. Melvin Montemerlo of NASA Headquarters, Dr. Michael Shafto of NASA 
Ames Research Center, Dr. Harold Hawkins of ONR, and Dr. Daniel Ling of IBM for their generous 
support of our meeting. We are also grateful to Dr. Stephen Ellis for his help in preparing this NASA 
publication. 


Nat Durlach and Tom Sheridan 
Conference Chairpersons 
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INTRODUCTION 


In a teleoperator system (see the accompanying figure), the human operator senses, moves 
within, and operates upon a remote or hazardous environment by means of a slave mechanism (a 
mechanism often referred to as a teleoperator). Signals in this environment are sensed by devices on 
the slave mechanism and communicated back to a human-machine interface and displayed to the 
human operator; the human operator’s responses are sensed by devices at the interface and commu- 
nicated back to the slave mechanism and used to control it. Teleoperators have been in use since the 
first nuclear laboratory experiments in the late 1940’ s. Teleoperation in the deep ocean, in space, in 
mines, and other remote, often hazardous, environments soon followed. Applications in medicine, 
involving exploration and operation within the human body, are now receiving increased attention. 
Early teleoperators were directly controlled by the operator in position (using isomorphic master- 
slave manipulators ) or in rate (where joystick position determined manipulator or vehicle rate). More 
recently, computers have intervened, and the operator has become a supervisor, issuing subtask goal 
and procedural commands and letting the teleoperator’s own computer then implement those 
instructions by closing a control loop through its own sensors and actuators. We term this more 
sophisticated teleoperator a telerobot and the mode of control supervisory control . General discus- 
sions of teleoperation are available in Johnsen and Corliss, 1971; Vertut and Coiffet, 1986; and 
Sheridan, 1989. 

In a virtual-environment system (again see the accompanying figure), the interactive human- 
machine interface is retained but the slave mechanism and its environment are replaced by a com- 
puter simulation. Video is replaced by computer graphics. The auditory and force sensations 
imparted to the human operator are similarly computer-generated. Since the rules governing the 
behavior of the simulation are under the control of the system designer, the operator in such systems 
can be presented with totally artificial worlds (e.g., having new physical laws). In contrast to a tele- 
operator system, where the purpose is to extend the operator’s sensorimotor system in a manner that 
facilitates exploration and manipulation of the physical environment, in a virtual-environment sys- 
tem, the purpose is to train, inform, alter, or study the human operator or to modify the state of the 
computer and the information environment. A major application in which the human operator is the 
target (because the goal is training) is that of flight simulation. Although flight simulators have been 
around for more than a decade, they had little impact outside aviation presumably because the appli- 
cation was so specialized and so expensive. Clearly, what has created the recent flurry of interest in 
virtual environments is the great reduction in cost and increase in power of computer-graphic work- 
stations, augmented, of course by miniaturization of sensors and display devices. Now some of the 
main applications are occurring in the entertainment industry (e.g., video games). Soon, we expect, 
there will be an increasing emphasis on designing virtual-environment systems specifically for sci- 
entific study of sensorimotor performance and for behavior modification by biofeedback. Applica- 
tions directed more towards changing the information environment than the human operator involve 
use of the system for scientific visualization and theoretical modelling (as in work on molecular 
docking). Also, virtual environment systems are now being incorporated within teleoperator systems 
in cases for which planning modes or predictive model outputs are required. The idea in this case is 
for the operator to steer the teleoperator through simulated operations presented computer- 
graphically just before performing the real operations. Alternatively, computer-graphic predictions of 
trajectory (bases on past and present control inputs) can be displayed automatically while the 



operator flies the vehicle or guides the manipulator . Such model-based predictions can then be 
superposed on the teleoperation video. General background on virtual-environments can be found in 
Sutherland, 1965; Krueger, 1983; Bolt, 1984; Haber, 1986; Foley, 1987; Fisher etal., 1988; Chung 
et al., 1989; Lanier, 1990; and Brooks et al., 1990. 

In either a teleoperator or virtual-environment system, when the sensory feedback can be made 
sufficiently compelling (through the use of multimodality displays in which the output is influenced 
by the motor behavior of the human operator) to make the operator feel present at the remote or 
imaginary site, the system is said to have telepresence. The extent to which telepresence is 
important for achieving good performance is itself an interesting research topic. 

In both teleoperator and virtual-environment systems, the human operator is effectively trans- 
formed into a new creature that is present in a new interactive environment. Also, in both cases the 
characteristics of the proposed transformation, as well as the sense of presence (or telepresence) in 
the new environment, are strongly influenced by the human-machine interface. Although there are 
some significant differences in the interface issues associated with the two types of systems (for 
example, in a teleoperator system it may be impossible to code certain targets independently because 
of inadequate resolution in the sensory system of the slave mechanism), there is clearly much in 
common. It is these common elements, namely, the transformed presence and the human-machine 
interface, that constitute the primary focus of this conference. 


Nat Durlach and Tom Sheridan 
Conference Chairpersons 
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I was asked to speak about how knowledge of visual perception could be applied to achieve tele- 
operation and virtual worlds. After a little thought I decided that reversing subject and object makes 
the question more interesting and provocative. The practice of teleoperation and virtual world tech- 
nology challenges our understanding of perception, and in the following I try to explain what I mean. 
But first let me say that vision science incorporates a vast amount of useful knowledge about the 
capabilities of the visual system. Fortunately, most of it is well-codified in the handbooks, of which 
the most recent and complete is Boff, Kaufman, and Thomas’s Handbook of Visual Perception mid 
Performance (1989). Unfortunately, this codified knowledge does not prepare us for the surprises 
attendant on the sensory and motor transformations effected by the new technologies. Instead, the 
surprises refer us to much less well understood areas. Hence the challenge to perceptual science from 
technology. 


World Regularities and the Habitual 


New technologies make it easy to transform spatially, temporally, and dynamically the relations 
among sensory inputs and motor outputs. For example, nothing could be simpler than making your 
hand appear to reach to the left when you intended it to reach to the right, to make you step up when 
you should step down, make the usually stable visible world appear to move as you move, and so on. 
The consequences can be errors in performance, disorientation, and a form of malaise like motion 
sickness. These examples make us realize to what extent our perceptions are dependent upon the 
regularities and habitual circumstances of our normal world and the dynamics of our actions in it. 
We are preadapted to these ordered states in multiple ways that are revealed by the effects of apply- 
ing the new technologies to the human-machine interface. Compared to our knowledge of basic 
visual processes, our understanding of these orderings is limited but not insignificant. Some of it has 
been relegated to the fringes of the field as aberrant and bizarre forms of perception. The following 
table enumerates several types of transformation and examples. 


Type 

Spatial Incongruity 
Temporal Asynchrony 
Altered Dynamics 
Deprivation 


Examples 

Japanese Illusion 
Sensory Rearrangement 

Phase Shift 
Time Delay 

Microgravity 
Force Amplification 

Blank Visual Field 
High Noise Level 


5 *f 
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Some of these non-habitual conditions may cause initial errors and disorientations. Some show 
adaptation during continued exposure i n the form of error reduction, better orientation, and reduction 
of motion sickness. Adaptation requires maintenance of a transformed set of regularities. If the trans- 
forms are complex and/or if the coupling is noisy, adaptation may be either very slow or not possible 
at all. However useful adaptation may be, it does take time and represents an additional load on the 
processing capacity of the system, sometimes called a cognitive load. 

Telepresence and Presence 

A particularly intriguing problem, related to the above-mentioned considerations, is the creation 
of telepresence. It raises the question of what conditions specify the sense of presence — a question 
that seems at first blush so obvious as to require no thought. It is the first axiom of perceiving, 
namely, that the observer be there to perceive. It implies a here and now spatial juxtaposition 
(confrontation) of object and observer. We can begin to identify conditions for the sense of presence 
as: 

1. Independent perception of an observer and an object external to the observer. 

2. Covariation of sensed observer movement independent of the sensed object yields the sense 
of an object external to the observer. 

3. Observer has access to the sensed object at will. 

Achievement of these conditions is, in turn, related to the regularities and habitual circumstances 
discussed above. 
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| This talk is a brief introduction to the basic auditory abilities of the human perceiver with 
particular attention toward issues that may be important for the design of auditory interfaces. The 
importance of appropriate auditory inputs to observers with normal hearing is probably related to the 
role of hearing as an omnidirectional, early warning system and to its role as the primary vehicle for 
communication of strong personal feelings. (| i : ) V ; 


Several basic properties of the human auditory perceiver should be kept in mind when designing 
interfaces. The range and resolution of the perceiver is impressive in terms of frequency (three 
decades with sensitivity to changes of 0.2%) and intensity (110 decibels with sensitivity to changes 
of a fraction of a decibel at almost all levels) so that the standards for clean sounds are quite high. On 
die other hand, our memory for absolute stimuli is much more limited, particularly with respect to 
intensity. In an identification paradigm, perceptual systems are generally limited to about 7 ± 2 
intensity levels, so that the ability to identify is much more limited than the ability to differentiate. 
The notion of the critical bandwidth is also important in understanding auditory phenomena and 
abilities. In its crudest form, the critical band notion, which is consistent with a wide variety of data, 
is that the auditory system behaves as if it contained peripheral filters with bandwidths of about one- 
tenth of an octave. 


The issues that arise in the design of auditory interfaces are different according to whether the 
interface is an isomorphic representation of the acoustic environment with natural sounds and trans- 
formations or a non-isomorphic mapping from fundamentally different inputs. In the completely 
isomorphic case, the problem is conceptually straightforward but very difficult to realize. Although 
one simply has to recreate the appropriate acoustic stimuli (from a robot or from a simulation) at the 
ears of the human listener, there are substantial engineering challenges in reproducing or creating the 
appropriate sounds. In addition to the obvious requirements for excellent fidelity in the acoustic sys- 
tem, the echo and reverberation effects on each sound must be appropriate and, critically, the motion 
of the listener’s head must be coupled to the robot or program. The complexity of the problem can be 
illustrated by thinking of the effect on the sounds received at the ears by several sound sources in a 
normal environment. In this case, as the head moves, the “head-related transfer function” of each 
source changes in a way that can only be specified with knowledge of the position of the source and 
the position and orientation of the head. The processing of the human perceiver imposes important 
constraints on the quality of the reproduced sound signal. 


In the simplest non-isomorphic cases, such as a scaled robot or a normal-sized robot in an envi- 
ronment with acoustic properties different than air (e.g., a helium rich environment), there are differ- 
ent problems, including the translations of signals to signals appropriate for human listeners. For 
example, a very small robot would have very small interaural differences that could be essentially 
undetectable to the human perceiver with an associated loss of ability to judge the azimuth of a 
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sound source. If these signals are processed to provide an appropriate stimulus on a human scale, the 
influence of the environment on the signals would have to be transformed and this would require 
significant knowledge of an environment that may be unfamiliar to the robot or its controller. The 
transformation of the received stimuli to provide appropriate acoustic inputs for a natural (human 
sized) situation requires a relatively full knowledge of the sound field. 

In the case of more extreme deviations from a humanoid robot or simulation, the issues become 
conceptually difficult to think about. For example, if the robot has multiple ears so that a more 
sophisticated picture of the acoustic environment could be computed by the interface, the mapping of 
the signals or the information to the two ears of the human pereeiver is difficult to design. Either 
some of the information will be neglected or we must learn how to increase the presentation of 
information to the human, possible by parsing the acoustic field and presenting multiple acoustic 
objects. This is a higher level recoding problem, and depends on techniques that have not yet been 
developed. Some of the attributes that result in the perception of separate acoustic objects are known 
and include separate spatial location, consistent modulation envelopes in frequency and amplitude, 
consistent harmonic structure, and others. . 

In some cases, if a stimulus is to be perceived as being generated by the listener, then there are 
special problems due to the fact that the sound is being received over two pathways, by the air-con- 
duction pathway and by the bone-conduction pathway. The air-conducted sound can be monitored 
and transformed appropriately by the interface, but the bone-conduction pathway is difficult to can- 
cel so that the stimulus perceived by the subject is a combination of the bone-conducted sound and 
the air-conducted sound provided by the interface, Learning would be a problem here if the new 
environment was trying to simulate an environment with light or heavy gases for example. In this 
case, the auditory stimulus would be a combination of the higher-pitched sound and the normal- 
pitched sound. * 

There are many cases in which unnatural stimuli are more effective than naturally occurring 
stimuli. [The argument that the human sensori-motor system is optimized by evolution (often applied 
to speech signals) cannot be applied in many cases because the constraints of the system are so 
complex and even difficult to know. In addition to questions of acoustics, for example, one can ask 
how important is the eating function to the design of the mouth or the necessity of being bom to the 
size of the head?] When there is a well-defined task to be performed, it is often advantageous to 
provide non-natural processing of the stimuli. For example, angle resolution near the midline can be 
improved significantly by presenting nonlinearly processed stimuli that would increase the interaural 
differences even though some distortion would be added that could be important for some stimuli. 

On the other hand, when the tasks are extremely varied or unpredictable, there are significant advan- 
tages to using natural stimuli and allowing the experience of the listener to be used to primary advan- 
tage with little training required for many tasks. In addition, there are situations in which an unnatu- 
ral stimulus may be advantageous such as the design of hearing aids with expanded ranges of levels 
and frequencies or in the design of auditory displays for non-auditory information. 
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Increasingly complex human-machine interactions, such as in teleoperation or in virtual envi- 
ronments, have necessitated the optimal use of the human tactual channel for information transfer. 
This need leads to a demand for a basic understanding of how the human tactual system works, so 
that the tactual interface between the human and the machine can receive the command signals from 
the human, as well as display the information to the human, in a manner that appears natural to the 
human. The tactual information consists of two components: (1) contact information which specifies 
the nature of direct contact with the object; (2) kinesthetic information which refers to the position 
and motion of the limbs. In this talk, we shall mostly be concerned with contact information. 


A/ 


A wide variety of tactile perceptions that we distinguish can be thought of as combinations of 
two classes of “primitives”: mode of contact (static indentation, relative tangential motion or slip, 
and vibration) and object properties (surface microtexture, shape, and compliance). It is plausible 
that by understanding the peripheral and central processing involved in each pairwise combination 
among the primitive classes, the processing of more complex natural stimuli can be unravelled. Our 
knowledge of cutaneous information processing is mostly based on human psychophysics and elec- 
trophysiological recordings from monkey afferent fibers. These fibers are of four types: Two types 
of slowly adapting fibers (SA I and II) that are associated with Merkel cells and Ruffini endings, and 
are responsive both when an object in contact is moving against the skin as well as under steady 
indentations; Two types of rapidly adapting fibers which respond only when the skin is moving, one 
type (RAs) terminating in Meissner corpuscles, and the other (PCs) in Pacinian corpuscles. When a 
probe indenting the skin is vibrated, lowest response threshold amplitudes for S As are at frequencies 
of 0-10 Hz, for RAs at 20-50 Hz, and for PCs at 100-300 Hz. Summarized below are some of the 
known results on the detection of slip and microtexture, shape, and compliance. 


Humans cannot detect the slip of a smooth glass plate on the skin; existence of detectable fea- 
tures on the surface is necessary. However, surprisingly small features (for example, 2 microns high 
dot causes RAs to respond and 0.06 microns high grating causes PCs to respond) on smooth surfaces 
are detected by humans and lead to the detection of slip of these surfaces, with the geometry of the 
microfeatures governing the associated neural codes. The division of labor among the different types 
of fiber populations in signaling the different events on the skin is clear-cut: S As signal the direction 
of skin stretch and hence the direction of impending slip; RAs and PCs signal the occurrence of slip 
with spatiotemporal or intensive codes, depending on whether the microfeature is a local one on a 
smooth background, or is distributed on the surface, respectively. When the surface features are of 
sizes greater than the response thresholds of all the receptors, redundant spatiotemporal and intensive 
information from all three afferent fiber types is available for the detection of slip. 

Among the different possible geometric representations of the shape of objects, the intrinsic 
description, i.e., the surface curvature as a function of the distance along the surface, seems to be 
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relevant for tactile sensing. Recordings of afferent responses to diverse shapes show that the depth of 
indentation and the change in curvature of the skin surface are represented in S A responses; in addi- 
tion, the velocity and the rate of change in skin surface curvature are represented in both S A and RA 
responses. The primary reason for such neural encoding is the form of the spatial variation of the 
pressure imposed by the object on the skin surface: Pressure peaks occur where the depths of inden- 
tation and/or changes in the skin surface curvature are high. The skin effectively acts as a low-pass 
filter in transmitting the mechanical signals, and the mechanoreceptors respond to the blurred ver- 
sions of the surface pressure distribution, thus encoding the shape of the object in terms of its surface 
curvatures. 

Compliant objects can be of two types: (1) those with a rigid surface (such as a piano key); 

(2) those with a deformable surface (such as cheese). By conducting psychophysical experiments 
under both active and passive touch, with or without local anesthesia that blocked the contact infor- 
mation, the following conclusions were reached. Humans are very good at discriminating softness of 
objects with deformable surfaces, and the discrimination is based on contact information. For com- 
pliant objects with rigid surfaces, discrimination of compliance is reliable only under active touch, 
when both contact and kinesthetic information is available, and the discriminability is poorer than 
that for objects with deformable surfaces. Preliminary neurophysiological data show that mostly SA 
and to a lesser extent RA responses provide the basis for our ability to discriminate compliance. 
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THE DISPLAY OF TACTILE INFORMATION 

Carl E. Sherrick 
Princeton University 
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There are a number of examples of natural tactile displays that can give us some insights about / - 
the solid geometry of touch, and recent experimental work on the subject has extended our thinking 
considerably. The concern of this conference is, however, more with synthetic or artificial displays 
for the production of a virtual environment. The majority of the talk will dwell on the features of 
synthetic displays that have enjoyed some success in one of two enterprises: The study of the spatio- 
temporal dimensions of stimuli that afford accurate and rapid processing of environmental informa- 
tion, or the use of displays in the design of sensory aids for disabled persons. 




Examples of synthetic displays are given, with some discussion of their application, success, and 
reasons for their failure, i.e., their current disuse. In the past, displays have been mainly of the type 
called diffuse, i.e., spread over a large area of the body. In more recent years, the improvements in 
electronic technology have made possible the production of more and more dense arrays of tactile 
stimulators, some of which, like the Optacon of Telesensory Systems, Inc., were developed for 
applied purposes, and now are used in basic research as well. 


This conference is best served by my talk if I list some of the things that have been learned about 
the, spatio-temporal properties of tactile displays that could be useful in the design of displays for 
teleoperator systems^ One of these has to do with the site of the display, which is commonly of con- 
cern to sensory aids developers for reasons of comfort and appearance. In addition to such concerns, 
however, it has become clear that issues of efficiency of energy usage, related to the thresholds for 
feeling, and the needed dynamic range of such devices, are also important to the designer, and these 
parameters vary greatly with the site of stimulation. Of equal importance is spatial acuity, i.e., the 
discrimination of high spatial frequencies on the skin, which varies systematically with the location 
of the stimulus array. 


There is a considerable literature dealing with such space-time phenomena as masking, real and 
apparent movement on the skin, temporal integration of both energy and coded information, salta- 
tion, and lateralization. These will be discussed in some detail as useful boundary conditions for 
device design. 

Another class of phenomena exist that may appear at first to be too esoteric for applied consid- 
eration, but which, because they have only a short history of systematic research, deserve considera- 
tion for their potential in enriching the perceptual field of the tactile display. One of these is the 
interaction of touch with other modalities, and a good example is found in studies of the temperature 
sense. Another is the phenomenon of distal attribution, which involves the transference of the display 
properties from the veridical site of stimulation to the object being simulated by the display. 
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A final word will be said about the importance of the learning process in acquiring a little-used 
perceptual-motor skill, which teleoperator systems must demand of the user when an unusual source 
such as touch is incorporated into the information stream. 
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NEURAL MECHANISMS OF KINESTHESIA 

Francis J. Clark 
University of Nebraska /; o f <r y JlV 

Omaha, Nebraska /// ' 7 

Our primary interest is in where the signals for position and movement of the limbs originate and 
how the information is encoded. Humans can independently sense the positions and movements of 
their limbs in the absence of vision, and these senses are derived from mechanoreceptors located in 
the limbs. Receptors in muscles, skin and joints are potential sources of position and movement sig- 
nals, but how (and whether) each of these groups contributes to kinesthesia remains uncertain. 

Recent evidence indicates that a sense of static-position derives from length receptors in muscle, 
whereas receptors in the skin, and possible though not likely the joints, can signal movement of a 
limb but not its static position. 

We distinguish this static-position sense from a movement sense on the basis of whether a sub- 
ject’s ability to sense a small displacement (3 to 5 deg) of a joint varies with the rate of joint dis- 
placement. Therefore, if subjects possess an awareness of the static-position of a joint, slow rates of 
rotation should not seriously degrade their ability to sense displacements. In the absence of a static- 
position sense, subjects would rely on movement signals that do depend on the rate of rotation and 
their ability to detect displacements should decrement with decreasing rate. This is roughly analo- 
gous to distinguishing a low-pass filter (a static-position sense) from a high-pass filter ) a movement 
sense only) by their frequency response characteristics. 

The interphalangeal (IP) joints of the fingers (and probably the toes) appear to lack a static-posi- 
tion sense. Subjects were unable to detect large displacements of the IP joints if the displacements 
were slow enough (our tests used rates less than 1 deg/min). In contrast, small displacements of the 
metacarpophalangeal (MCP) joint of the finger could be detected at the slowest rates we used. This 
is a surprising finding given that people seem quite unaware they have this kinesthetic deficit at the 
fingers. 

At present, we are examining the consequences of not having a static-position sense at the fingers 
apart from an inability to detect very slow displacements. Using a more demanding test, we have 
found no substantial difference in the accuracy of reproducing target positions passively imposed on 
the MCP joint and the proximal interphalangeal (PIP) joint of the index finger (the MCP has and the 
PIP lacks a static-position sense). Our working hypothesis is that subjects normally use movement 
sense mechanisms to detect displacements, probably because they are faster, but how (or whether) 
subjects manage to obtain an absolute position reference with only a movement sense available 
remains unknown and is an issue under investigation. 

We have also shown that subjects can reproduce target positions of a joint from memory about as 
accurately as they can match the position of the joint to its corresponding joint on the opposite limb. 
Reproducing targets from memory remains remarkably accurate even after 24 hours. The existence 
of a stable, long-term memory for limb position is an important component of a model for motor 
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control we are investigating. Irrespective of the mechanism, the accuracy of this memory compli- 
cates the study of position and movement senses. For example, one cannot study a static-position 
sense by moving a joint to some position and waiting until all movement signals have faded away. 
With a memory, one could not distinguish a true static-position mechanism from a mechanism that 
derived position from a velocity signal and stored the result in a memory. 
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ANALYSIS OF THE HUMAN OPERATOR SUBSYSTEMS 

Lynetve A. Jones and Ian W. Hunter 

McGill University $ MX ' l>/ 

Montreal, Quebec, CANADA ^ 

Except in low-bandwidth systems, knowledge of the human operator transfer function is essential 
for high-performance telerobotic systems.. This information has usually been derived from detailed 
analyses of tracking performance, in which the human operator is considered as a complete system 
rather than as a summation of a number of subsystems, each of which influences the operator’s out- 
put. Limitations in analytic techniques and insufficient knowledge about the characteristics of the 
human operator have resulted in these “black box” depictions of tracking performance. One conse- 
quence of this approach is that it has been impossible to determine the source of change in the 
parameters of the transfer function under different conditions. Studies of one of these subsystems, 
the limb mechanics system, demonstrate that large parameter variations can occur (e.g., the elastic 
stiffness of a limb can change over a 50-fold range for different levels of muscle activation), that can 
have a profound effect on the stability of force-reflecting telerobot systems. 
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One of the objectives of our research is to decompose the performance of the human operator 
system (i.e., its damping, natural frequency, pure delay, static gain) in terms of the subsystems 
depicted above, in order to establish how the dynamics of each of these elements influence the 
operator’s responses. For example, the delays associated with each subsystem are different (ranging 
from 15ms for contractile dynamics to over 150ms for cognitive processing), and for some subsys- 
tems the transformations are linear and time invariant, whereas for others they are nonlinear and time 
varying. Recent advances in nonlinear system identification make it possible to characterize the 
dynamics of the human operator subsystems in a single experimental session. Ideally the same 
equipment and analytic procedures should be used in all phases of this process. On die basis of the 
results from such an analysis, it should be possible to predict for any given operator the conditions 
(e.g., muscle-joint segment, type of controlled response, mechanical properties of the human 
interface) under which control is optimal. 
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With this objective in mind, we have constructed an apparatus for studying the human neuro- 
muscular system in terms of its neural (contractile dynamics), mechanical (limb mechanics), percep- 
tual (sensory thresholds), and tracking (human interface) characteristics. The apparatus consists of 
two powerful computer-controlled linear actuators instrumented with displacement and force trans- 
ducers. With this equipment the neuromuscular function of the muscles controlling the elbow can be 
determined by imposing small, wide-bandwidth, stochastic length perturbations to the forearm. In 
addition the mechanical, reflex, and neural excitation to force dynamics of the neuromuscular system 
controlling the elbow flexor/extensor muscles can be measured, sensory thresholds can be calculated, 
and finally the tracking dynamics of the human operator control system can be determined. To date, 
studies of these subsystems have been limited to single joints such as the elbow, and the effects on 
tracking performance of varying the response type and human-interface properties have been 
investigated. 

The characteristics of the human operator in a pursuit-tracking task change as a function of the 
operator’s response. When subjects track a visually presented target either by changing the position 
of their forearm or by modulating the forces generated by the elbow flexor and extensor muscles, 
force control is superior to position control in terms of response delay (100 ms less), but subjects are 
more accurate when position is the controlled variable. Position control is, however, influenced by 
the mechanical properties of the human interface, and in particular changes in manipulator stiffness 
affect both response delay (less at higher [up to 2000 N/m] stiffness amplitudes) and tracking accu- 
racy (more accurate at lower [< 1500 N/m] amplitudes). 
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INTERSENSORY COORDINATION >7 ^ 

J R L kn 
J. R. Lackner 

Brandeis University 
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Perception of body orientation and configuration is dependent on multiple sources of afferent and 
efferent information about the spatial configuration of the body and its relationship to the support 
and force characteristics of the surroundings. Such information allows us under normal circum- 
stances to preserve an accurate and stable distinction between those changes in receptor activity con- 
tingent on self-motion and those contingent on motion of or within the environment. Maintenance of 
this distinction is essential for the ongoing control of normal body movement and posture. 

Because of this active interrelating of sensory and motor cues whenever there is an error in the 
registration of ongoing body orientation, auditory and visual mislocalizations of related magnitude 
and time course can occur. As a consequence, depending on the perceived spatial representation of 
the body, identical patterns of arrival time and intensity cues at the ears (and identical retinal and 
oculomotor signals) can give rise to the perception of sounds (and visual objects) in widely disparate 
spatial positions in relation to the head and to the external environment. The interaction between sen- 
sory localization and the representation of body orientation is actually bidirectional. For example, a 
rotating sound field can elicit illusory self-rotation and compensatory eye movements in a stationary 
listener. 

The conscious appreciation of limb position is influenced by muscle afferent signals. When the 
body or tendon of a muscle is mechanically vibrated, for example at 120 Hz, the muscle reflexly 
contracts. This contraction is.known as a tonic vibration reflex (TVR). If the motion of a limb mov- 
ing under the action of a TVR is resisted, illusory motion of the (unseen) stationary limb will be 
experienced. For example, if die biceps brachii muscle is vibrated, the restrained forearm will be felt 
to move into extension. A target light attached to the stationary index finger of the hand, in an 
otherwise dark room, will be seen to move in keeping with the apparent motion of the arm even 
though the eyes maintain steady fixation. If the subject is allowed to see his index finger in relation 
to the surroundings, but not the rest of his forearm, he will see his finger in one spatial location and 
feel the rest of his arm to be in another spatially distinct location. 

During body movement in a high force field, the relationship between muscle activity and mus- 
cle spindle feedback is altered, much as in die case of the vibration illusions just described. If an 
individual does deep knee bends during exposure to twice earth gravity, then he will misperceive the 
extent of his own voluntary motion and simultaneously experience visual motion of his surroundings 
by the amount of that misperception. This demonstrates that there is a dynamic relationship between 
intended movements and the actual movements executed, and the perception of those movements. 

During normal terrestrial locomotion, deformation of the optic projection of the visual world, the 
intensity and timing of efferent signals and of muscle afferent signals, and the distribution of contact 
forces of support on the body co-vary and are mutually corroborative. If the relationship between 
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voluntary stepping movements, body displacement, and visual flow are artificially altered, it is pos- 
sible to distort the perception of voluntary actions so that an individual voluntarily stepping forward 
can perceive himself to be voluntarily walking backwards, or to be stepping more slowly or more 
rapidly than he actually is. Common to all the remappings elicited are combinations of perceived 
self-motion, perceived limb activity, seen motion of the visual world, and felt motion of the ground 
that are dimensionally consistent (even if not physically probable, such as changes in apparent leg 
length) with minimal departures from terrestrial constraints. 
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WATCH WHAT YOU SAY, YOUR COMPUTER MIGHT BE LISTENING: 

A REVIEW OF AUTOMATED SPEECH RECOGNITION 

Stephen De Gennaro 
IBM Thomas J. Watson Research Center 
Yorktown Heights, New York 

Spoken language is the most convenient and natural means by which people interact with each 
other and is, therefore, a promising candidate for human-machine interactions. Speech also offers an 
additional channel for hands-busy applications, complementing the use of motor output channels for 
control. 

? ■ 

Current speech recognition systems vary considerably across a number of important characteris- 
tics, including vocabulary size, speaking mode, training requirements for new speakers, robustness to 
acoustic environments, and accuracy. Algorithmically, these systems range from rule-based tech- 
niques through more probabilistic or self-learning approaches such as Hidden Markov Modeling and 
Neural Networks. 

This tutorial begins with a brief summary of the relevant features of current speech recognition 
systems and the strengths and weaknesses of the various algorithmic approaches. 

Issues critical to the successful application of speech recognition in human-machine interactions 
will be discussed in more detail. These issues include: “raw” recognition vs. speech understanding, 
grammar-based, constrained dialog vs., natural-language, free discourse and the imposition of 
appropriate constraints to make the task technologically feasible, with acceptable accuracy. The 
potential delay in recognition, due to algorithmic features or to interactive verification of recognized 
words for error correction, must also be considered. The need for verification is determined by both 
the expected error rate and the cost of each error. The cost function could be dynamic if errors will 
have significantly different impact given the state of the system. 

As in any human-machine interface, not all of the constraints are due to the “-machine” half of 
the interaction; the “human-” partner also imposes a share of burdens. For example, people will not 
always say exactly what is expected of them, even when carefully prompted with a small set of pos- 
sible choices. Stress, attention, and general physical state can have strong effects on system perfor- 
mance. 

For applications that are not currently speech-driven, it might also be necessary to reformulate 
the basic human-machine interface. For example, moving a cursor over a document by voice in the 
most straightforward way (up-up-up-left-left- ...) is not as efficient (and probably not as acceptable) 
as moving the cursor by keys or mouse. Recasting the basic problem as “go to the word HELLO is 
the second paragraph” provides a better match between the task and speech control. Navigating a 
robot by voice instead of by joystick might require similar reformulation. Of course, while “go to the 
word ...” is more natural, it imposes new requirements to understand the command and map it into 
the appropriate action. 
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Given these characteristics, it is not yet reasonable to expect that an off-the-shelf speech recog- 
nizer can be used as a transparent replacement for a keyboard or joystick as input to any arbitrary 
application. The highest performance levels will be achieved when the application and recognition 
processes are more tightly coupled. 

Using existing speech recognition technology, it is certainly possible to build successful virtual 
environments. This is particularly true for systems that mimic environments in which speech is 
already the primary communication mode, and where the control “protocols” have been well 
defined. For example, Air Traffic Control training consoles have been demonstrated using continu- 
ous speech, but the task is constrained to deal only with objects (i.e., specific flights) on the trainee’s 
console to provide reasonable accuracy and response delay. A general problem with coupling, how- 
ever, is that it leaves the system vulnerable to unanticipated scenarios, with concomitant risk that the 
system will not respond gracefully to “exceptional” interactions. 

This tutorial will conclude with some pointers to conferences and publications that reflect the 
current state of commercially available speech recognizers, and the state of research in this field. 
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EYE MOVEMENT CONTROL AND TOP-DOWN SCANPATH VISION AS A 
DESIGN METAPHOR FOR ROBOTIC VISION AND CONTROL 

Lawrence W. Stark -5 l# o 

University of California , ^ 

Berkeley, California 

I. Eye Movement Control 

1 . Kinetics: Time Optimal Control of Saceades 

Eye movements are elegant biological examples for application of time-optimal engineering 
control theory. The muscles that drive eyeball rotations are very long and strong and fast. The 
eyeball has low rotational inertia so that the internal viscosity of active muscle dominates plant 
dynamics; thus we may think of the system as a visco-elastic load. What controller signal shapes 
would produce a time-optimal movement? First, reciprocal innervation, proposed by Descartes 
(#149) and experimentally found by Sherrington, should be observed (Figure 1 A). Next, a set of 
multipulse-step control signals first described by Clark and Stark (#137) can be justified on theoreti- 
cal grounds by Bellman-Pontriagin bang-bang theory. 

2. Dual Mode Control: Sampled Data Operator 

Another beautiful example is the sampled data control model of Young and Stark (#27) that 
was experimentally verified by study of transient responses in normal subjects. A further exciting 
demonstration was found in the explanation of “macro-saccadic oscillation” found in patients with 
cerebellar disease that effectively removed the continual adaptive gain adjustment and left the patient 
with high gain instability oscillation (#156). (Figure IB). 

n. Higher Level EM Control and Scanpath Theory 

3. Active Looking and the Kantian Theory of Perception 

A more metaphoric theory, the Scanpath Theory has been developed by Noton and Stark 
(#1 10) and Stark and Ellis (#21 1) to account for the repetitive sequence of saceades that enable the 
eye and its fovea centralis to traverse over important sub-features of a picture or scene. This has been 
taken to mean than an internal cognitive model drives active looking in a top-down manner. The evi- 
dence to support this theory has to do with statistical proof that there is indeed repetition of scanpath 
sequences. Further evidence comes from studies of ambiguous figures wherein the same subject 
looking at the same picture switches both mental states of perception and patterns of repetitive sac- 
cadic sequences. Most recently, Brandt and Stark (in preparation) have used visualization tasks. A 
subject recalling a previously seen diagram demonstrates scanpath movements similar to those 
looking at the earlier viewed diagram; here only the internal model can be responsible for the scan- 
path sequence and, indeed, string editing distances support this finding quantitatively. (Figure 2A). 
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The philosopher, Kant, reasoned that internal representations (Platonic ideals) controlled active 
perception in a top-down procedure that could then organize sensation (without time and without 
space) received from the chaotic work of appearances. In this view, the virtual model within a human 
brain is more real than the external world of the naive realistic! (Figure 2B). 


Acknowledgment to: NASA Ames Research Center Cooperative Agreement NCC-286, Dr. Stephen 
Ellis, Technical Monitor, and to California Institute of Technology, Jet Propulsion Laboratories, 

Dr. Anatal Bejczy, JPL Technical Manager, for their support. 
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III. Top-Down Robotic Vision 


4. Models of Robots and ROIs 

For our use of visual feedback, for use by autonomous algorithms, for use by manual operators, 
and for use by supervisory controllers. We have taken this top-down scheme from human vision and 
introduced it as a basic design principle in our Telerobotic Laboratory. Instead of searching for 
important features of the TRWE (telerobotic working environment) with bottom-lip image process- 
ing algorithms, we instead use a top-down approach. We already know where a particular robotic 
link is located in 3D space, we know where the cameras are pointed, we know what important 

parameters we need to measure. Thus, we use our knowledge in the form of forward-loop control 
model not only to control the robots and the cameras, but also to direct the image processing (#312 
and A304). We construct ROIs (regions-of-interest) for each camera frame so that the image 
processing can be done rapidly and robustly (Figure 3 A). 

Within the ROIs, local thresholding and centroid calculations enable us to compute a feedback 
model, indicated by crosses that represent the visual measurement of the location of key on-the- 
scene-visual-enhancements. This, of course, is done dynamically so that rapid feedback permits 
closed-loop control of the robotic movement, with the human operator acting as a supervisory con- 
troller only (Figure 3B). 

IV. Over- All Robotic Control Scheme 

5. Model of RWE 

The overall model (Figure 4A) permits one to appreciate the communication functions of our 
top-down image processing scheme. Communication is restricted to parameters of the visual models, 
an enormous bandwidth reduction compared to sending video pictures of a number of views of the 
TRWE. The supervisory controller looks not at the camera view of the scene, but rather at a model of 
the TRWE! Is this going to be satisfactory? (#T411, #313). 

6. Evaluation of Displays and HMD 

Several evaluation studies of visual display requirements have been carried out in our labora- 
tory (#301, #A313). Dr. Won Soo Kim and colleagues have established that, although for some pur- 
poses enhanced 2D displays enable the operator to perform satisfactorily, virtual 3D displays are 
robust to various deficiencies and enable performance at least as good as the best 2D displays. Also, 
models alone or models superimposed onto video pictures are better than video pictures for short- 
term robotic control tasks (Figure 4C, #A291). In another study. Dr. Greg Tharp and colleagues 
demonstrated that 5 to 15 Hz update rates for a stereo display were fast enough to permit the subject 
to slew his head freely without suffering a penalty in performance (delays were less than 1 frame 
interval). (Figure 4B). 
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V. Summary 

Telerobotics with virtual environments is an exciting and worthwhile field of research. It offers 
challenges to engineering design of displays, of control, and of communication schemes. It relates to 
human performance since the human operator is a component either as a direct manual controller or 
as a supervisory controller. It will teach us more about human capacities and we can use knowledge 
of human plans and schemas to produce initial designs. We are also learning more about ourselves. 
Did we thing about “presence” before we considered telepresence? 
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Figure 1 A. Descartes model. 

#149 Kenneth J. Ciuffreda and Lawrence Stark, “Descartes’ Law of Reciprocal Innervation,” American Journal of 
Optometry and Physiological Optics 52: 663-673 (1975) 
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Figure IB. SD clinical simulation. 

#27 Laurance R. Young and Lawrence Stark, “Variable Feedback Experiments Testing a Sampled Data Model for Eye 
Tracking Movements.” IEEE Transactions on Human Factors in Electronics HFE-4: 38-5 1 (1963) 


#156 John B. Selhourst, Lawrence Stark, Alfred L. Ochs and William F. Hoyt, “Disorders in Cerebellar Ocular Motor 
Control. II. Macrosaccadic Oscillation: An Oculugraph, Control System and Clinico- Anatomical Analysis.” Brain 99: 
509-522 (1976). 
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SCANPATH THEORY 
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Figure 2A. Scanpath theory. 

#1 10 David Noton and Lawrence Stark, “Eye Movements and Visual Perception.” Scientific American 224: 334-43 
(1971). 

#211 Lawrence Stark and Stephen Ellis, “Scanpaths Revisited: Cognitive Models Direct Active Looking.” In: Eye 
Movements, Cognition and Visual Perception, ed. Fisher, Monty and Senders, New Jersey: Eribaum Press, 193-226 
(1981). 
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Figure 2B. 

#287 L. Stark, W.H. Zangmeister, B. Hannaford and K. Kunze, “Use of Models of Brainstem Reflexes for Clinical 
Research.” In: Clinical Problems of Brainstem Disorders ; Thieme Publishers, New York: 172-184 (1986). 
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Figure 3A and 3B. 

#312 Lawrence Stark, Barbara Mills, An Nguyen, Huy X. Ngo, “Instrumentation and Robotic Image Processing Using 
Top-Down Model Control.’' Robotics and Manufacturing, Jamshidi et al., eds., ASME, NY (1988): 675-682). 


#A3Q4 An Nguyen ami Lawrence Stark, “3D Model Control of Image Processing.” NASA Conference of Space 
Telerobotics, Pasadena (1989). 



Figure 4A. 

#313 Lawrence Stark, et al., ‘Telerobotics: Problems and Research Needs.” IEEE Transactions, Aerospace and 
Electrical Systems 24: 542-551 (1988). 

#T41 1 Lawrence Stark, “Biological Redundancy,” in progress. 
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Figure 4B. 

#301 Won Soo Kim, Frank Tendrick, Stephen Ellis and Lawrence Stark, “Visual Enhancement in Pick-and-Place Tasks: 
Human Operators Controlling a Simulated Cylindrical Manipulator.” IEEE Journal of Robotics and Automation RA-3: 
426-436 (1987). 

#A313 Gregory Tharp, Andrew Liu, Hitomi Yamashita, Lawrence Stark, Brenda Wong and Jurgen Dee, “A Helmet 
Mounted Display to Adapt the Telerobotic Environment to Human Vision.” Johnson Space Center, Texas (1989). 



Figure 4C. 

#A291 Kim, W. S., Takeda, M. and Stark, L., “On-the-screen Visual Enhancements for a Telerobotics Vision System.” 
Proceedings IEEE International Conference, Systems, Man & Cybernetics, Beijing (1988). 
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TELEROBOTIC REMOTE PRESENCE: ACHIEVEMENTS AND 

CHALLENGES 

j/o-py 

Walter A. Aviles O^J^y 

Naval Ocean Systems Center ' / 

Kailua, Hawaii ^ f'O 
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The Marine Sciences and Technology Department of the Naval Ocean Systems Center has as one 
of its particular thrusts the development of telerobotic work systems which blend the capabilities of a 
human with those afforded by automation technologies. The goal of these systems is to give a human 
operator the subjective experience of being at a remote work site when he is in fact at a location well 
removed from the (potentially hazardous) work area. Systems of this type are said to exhibit “tele- 
presence”. Telepresence is accomplished through the use of spatially correspondent controls and 
high-fidelity sensory feedback systems which utilize the human operator’s cognitive and sensory 
capabilities in a manner that is natural to him. A major advantage of the telepresence approach is that 
the human operator does not have to concentrate on how to use unfamiliar or unnatural displays and 
controls but instead uses his own cognitive and manipulative abilities to perform the task. As a con- 
sequence, the telepresence approach minimizes the time required to perform work as well as mini- 
mizing the amount of training required to use sophisticated undersea work systems. 

Although the goals of a telerobotic human-machine interface exhibiting telepresence are intu- 
itively straightforward to grasp, the quantification of terms such as “correspondent controls” and 
“high-fidelity sensory feedback” in a manner useful for development remains a major challenge to 
the field. In addition, the proper blending of automated assistance, remote work site sensory feed- 
back, and artificially generated sensory feedback, is both a fascinating and challenging endeavor. 
This talk will attempt to outline the progress in human-machine interfaces from the perspective of 
telerobotic remote presence and to suggest avenues for future research. 




jy/' o 



3 / 


N95- 14024 


ISSUES IN THE DESIGN OF HIGH DEXTERITY, FORCE REFLECTIVE 

TELEOPERATORS 


S. C. Jacobsen, E. K. Iversen, C. C. Davis, K. B. Biggers, and D. K. Backman 

University of Utah 
Salt Lake City, Utah 


The Center for Engineering Design at the University of Utah is developing an anthropomorphic, 
hydraulically actuated, teleoperated arm. The system includes a sixteen degree-of-freedom slave 
manipulator controlled by a kinematically identical, sixteen degree-of-freedom force-reflective, 
exoskeletal master. The project has focused on four areas: (a) formulating a realistic set of design 
objectives which balance, against technical realities, the desire for performance, reliability and econ- 
omy, (b) understanding control issues, (c) designing and fabricating new subsystems (presently 
unavailable) necessary for the construction of a successful machine and (d) integrating subsystems, 
through a series of prototype stages, into an operational teleoperation system. 

Generating a comprehensive set of design objectives required the consideration of fundamental 
questions such as: (1) what should be the kinematic configuration of master and slave, hand and arm, 
(2) how many degrees of freedom will be necessary for the level of dexterity desired, (3) should the 
master be exoskeletal, or some other configuration, (4) what is the importance of force reflection and 
how can its fidelity be improved beyond current systems, (5) what are the realistic actuation alterna- 
tives considering active and passive performance requirements, (6) what performance is required of 
sensor systems, (7) when necessary, can tendons be successfully used to remotize actuators. 

Our work in the control area focused on: (1) developing stable position and torque feedback 
approaches which result in desirable inter-system impedance (frequency dependent gain, stiffness 
and damping), (2) generating acceptable joint-torque compensation which removes undesirable 
effects of gravity, mass and viscous drag, (3) increasing system bandwidth to improve operator 
awareness of loads imposed on the slave. 

Extensive efforts were expended in the design and fabrication of presently unavailable subsys- 
tems such as: (1) rotary and linear hydraulic actuators along with required servo valves, (2) joint- 
torque sensors capable of remote-axis-sensing, high stiffness, low stiction and minimal sensing of 
loads on other axes, (3) position sensors with acceptable accuracy, (4) computation systems which 
include digital and analog electronics for implementation of servo, intermediate and higher level 
control functions, (5) systems for long range communication between the master and slave systems. 
Designing subsystems such that they “package” into the available shaped volumes as well as satisfy- 
ing other constraints was a problem of substantial magnitude. Packaging remains an underestimated 
problem which in fact dominates many design decisions. 

Finally, work is proceeding towards integrating subsystems into a working machine which 
actually achieves its objectives with respect to performance, reliability and economy. The system 
must: (1) be capable of operating in sea environments which includes corrosion and substantial 
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pressure, (2) survive substantial mechanical abuse and misuse, (3) include mechanical machine- 
operator interfaces which insure comfortable, nonfatiguing operation over long periods of time. 

The presentation will address the issues mentioned above as well as include a video demonstra- 
tion of the system executing selected dextrous tasks. 
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TELE-EXISTENCE AND/OR CYBERNETIC INTERFACE 

IN JAPAN 


N95- 14025 

STUDIES 


SusumuTachi 

Mechanical Engineering Laboratory, MITI 
Tsukuba Science City, Ibaraki, JAPAN 


Tele-existence aims at a natural and efficient remote control of robots by providing the operator 
with a real time sensation of presence. It is an advanced type of teleoperation system which enables a 
human operator at the controls to perform remote manipulation tasks dexterously with the feeling 
that he or she exists in one of the remote anthropomorphic robots in the remote environment, e.g., in 
a hostile environment such as those of nuclear radiation, high temperature, and deep space. 

Systematic research for the development of tele-existence has been conducted at MEL by feeding 
back rich sensory information which the remote robot has acquired to provide the operator with a 
real-time sensation of presence. 

With the advent of science and technology, it has become possible to envision teleoperation with 
a sensation of presence. The concept of projecting ourselves by using robots, computers and cyber- 
netic human interfaces is called Tele-Existence, and a lot of effort has been made for the realization 
of this concept. This concept also provides an extension of human sensing and muscular capabilities. 

The tele-existence system consists of intelligent mobile robots, their supervisory subsystem, a 
remote-presence subsystem and a sensory augmentation subsystem, which allows an operator to use 
the robot’s ultrasonic, infrared and other, otherwise invisible, sensory information with the computer 
graphics-generated pseudo-realisitic sensation of presence. In the remote-presence subsystem, real- 
istic visual, auditory, tactile, kinesthetic and vibratory displays will be realized (ref. 1). 

Using this system, a human operator can be in a safe and comfortable environment and, at the 
same time, be present or exist at other environments where the robots are working. He or she will 
monitor the work through the robot’s sensors, and, if necessary, conduct the task on behalf of the 
robot as if he was doing the work himself (ideally) or as if he was working inside the robot 
(practically). 

The principle of the tele-existence sensory display has been proposed. Its design procedure has 
been explicitly defined. Experimental visual display hardware has been built, and the feasibility of 
the visual display with the sensation of presence has been demonstrated by psychophysical experi- 
ments using the test hardware (refs. 1,2). 

A method has also been proposed to develop a mobile tele-existence system, which can be 
remotely driven with the auditory and visual sensation of presence (ref. 3). A prototype mobile tele- 
vehicle system has been constructed and the feasibility of the method has been evaluated. 


34 



In order to study the use of the tele-existence system in the artificially constructed environment, 
the visual tele-existence simulator has been designed, a pseudo-real-time binocular solid model robot 
simulator has been made, and its feasibility has been experimentally evaluated (ref. 4). 

An anthropomorphic robot mechanism with an arm having seven degrees of freedom has been 
designed and developed as a slave robot for feasibility experiments of teleoperation using the tele- 
existence method. An impedance controlled active display mechanism and a head mounted display 
have also been designed and developed as the display sub-system for the master. The robot’s struc- 
tural dimensions are set very close to those of humans, and it is designed to mimic the movement of 
humans (ref. 5). By using this tele-existence master slave system, the feasibility of the tele-existence 
master slave system has been demonstrated. 

REFERENCES 

1 . S. Tachi et al., “Tele-existence (I) - Design and evaluation of a visual display with sensation of 

presence-,” Proceedings of the 5th Symposium on Theory and Practice of Robots and 
Manipulators (RoManSy84), pp. 245-254, CISM-IFToMM, Udine, Italy, June 1984. 

2. S. Tachi and H. Arai, “Study on tele-existence (II) - Three dimensional color display With sen- 

sation of presence - ,” Proceedings of the “85 International Conference on Advanced 
Robotics (ICAR), pp. 345-352, Tokyo, Japan, Sept. 1985. 

3. S. Tachi, H. Arai, I. Morimoto, and G. Seet, “Feasibility experiments on a mobile tele-existence 

system,” The International Symposium and Exposition on Robots (19th ISIR), Sydney, 
Australia, November 1988, 

\« . - ••• ; . 

- it 

4. S. Tachi, H, Arai, and T. Maeda, “Tele-existence simulator with artificial reality (1), - Design 

and evaluation of a binocular visual display using solid models-,” Proceedings of the IEEE 
International Workshop on Intelligent Robots and Systems, pp. 719-724, 1988. 

5. S. Tachi, H. Arai, and T. Maeda, “Development of an anthropomorphic tele-existence slave 

robot,” Proceedings of the International Conference on Advanced Mechatronics (ICAM), 
pp. 385-390, Tokyo, Japan, May 1989. 


35 



N95- 14026 


* 1 / 2 -37 

S, O & 

42 


TELEROBOTICS IN THE DEEP OCEAN 


? 


\ 


Hagen Schempf and Dana R. Yoerger 
Woods Hole Oceanographic Institution 
Woods Hole, Massachusetts 


Remotely operated vehicles can be made more useful for a wide range of survey tasks through 
the use of precision navigation and automatic control integrated under a proper supervisory control 
framework. Tasks ranging from scientific survey to ship hull inspection can be done more produc- 
tively and produce a more useful result if the motions of the ROV are tightly controlled. A key ben- 
efit of automatic control is the ability to repeat a track at a later time to study dynamic processes 
seafloor. 

This paper presents experimental results of the control system for the JASON ROV that has been 
designed for precision survey and other automated applications* The JASON control system empha- 
sizes a form of supervisory control where the human pilot and the automatic system share the control 
task. Results presented included hovering, automatic track following, and several interactive modes. 

JASON is equipped with an underwater manipulator that can exhibit a wide range of compliance 
through a combination of mechanical design and software control is described and its performance 
characterized. The major goal of the design was to produce a manipulator that can control the inter- 
action fqrces with the work task and can work reliably in the hostile deep-ocean environment. The 
manipulator’s performance has been characterized in the lab and its overall operational utility has 
been confirmed during an archaeological excavation at 700 meters depth in the Mediterranean. 
Results from the lab tests and sea trials will be presented? » 
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MODELS OF REMOTE MANIPULATION IN SPACE 


Blake Hannaford 
University of Washington 
Seattle, Washington 



Astronauts operating in space can hardly be called autonomous. Their progress is constantly 
being monitored and supported by the ground team. The same procedures will apply to robotic sys- 
tems in space - no matter how intelligent. This is also true of other important manipulation tasks. For 
example, a surgeon must perform an operation within a system of supervision and documented pro- 
cedures which ensure quality care and accountability. What these two domains have in common is a 
high value environment in which a highly specialized manipulation is performed a small number of 
times (in contrast to factory automation). Thus robots involved in high value manipulation must be 
effectively coupled to a human operator either at the work-site or remotely connected via communi- 
cation links. 


Hundreds of hours of man in the loop experiments have been performed to quantify the perfor- 
mance of telemanipulation systems with different levels of capability. 

In order to make use of experimental performance evaluation data, models must be developed. 
Powerful models of remote manipulation by humans can be used to predict manipulation perfor- 
mance in several ways. First, they will allow prediction of performance in future systems based on 
today’s laboratory systems. In this paradigm, the models are developed from experimental data, and 
then used to predict performance in slightly different situations. Second, accurate telemanipulation 
models will allow design of manipulation systems which extend manipulation capability beyond its 
current bounds. 

The barriers which are currently being extended by state of the art remote manipulation systems 
include distance and scale. Distance barriers are most relevant to space telemanipulation although the 
large manipulators made feasible by the lack of gravity involve scale factors of about 10. Micro- 
teleoperators are now being developed which couple the operator to micro-worlds. For example, a 
force reflection system has recently been developed by IBM which couples the operator to the tip of 
a scanning tunneling microscope: a scale factor of about 10~ 7 ! 

The famous time delay problem inherent in overcoming the distance barrier is yielding to new 
methods in which control is shared between the manipulator and the human operator in ways which 
allow energetic interaction between manipulator and environment. 

Models developed by several laboratories to explain the dynamics of kinesthetic force reflection 
will be briefly reviewed and applied to the experimental data. Two-port network models have been 
successful at improving the understanding and performance evaluation of teleoperators with kines- 
thetic force feedback in which the human operator, master manipulator, slave manipulator, and envi- 
ronment are coupled together through servo control loops. Such systems can create a sense that the 
operator directly manipulates the environment. 
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A relatively new model of task performance based on Hidden Markov Models, is able to analyze 
progression of multi-stage compound tasks. The HMM has several properties that make it attractive 
for use as a near-real-time execution monitor and post hoc “explainer” in practical applications. 
Among them are: 

o Explicit Representation of Uncertainty 

o Support for “knowledge based” operation (the HMM can be constructed heuristically by an 
expert). 

o Closed form, thoroughly developed, efficient algorithms for state estimation (Viterbi) and model 
identification (Baum- Welch). 
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TELEOPERATING THE SPACE STATIONS ^ 7 

Marvin L. Minsky 

Massachusetts Institute of Technology 
Cambridge, Massachusetts 

(Talk not presented.) 
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SOME CRITERIA FOR TELEOPERATORS AND VIRTUAL 


ENVIRONMENTS FROM EXPERIENCES WITH 
VEHICLE/OPERATOR SIMULATION 
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Henry R. Jex 
Systems Technology, Inc. 
Hawthorne, California 


Over 30 years of experience in the simulation, modeling and analysis of man-machine control 
systems has resulted in a variety of rules-of-thumb and some quantitative criteria for assuring valid 
simulation of “real-world” situations and likely users. It is hoped that these criteria, which apply 
directly to teleoperations (i.e., no communication delays; operator-in-the-loop; no “smart-robot” del- 
egation), will be useful for designing better telerobots and in simulating telerobotic controls and vir- 
tual environments. 

First, some definitions are given: “Valid simulation” is defined as that producing the same opera- 
tor behavior, performance and mental workload as the real-world case. “Delay artifact” is the 
(measured) sum of all throughput delays from: sensing, computing, pipeline and display generation. 
“Kinetosis” is the vertigo and malaise from conflicts among visual, vestibular and proprioceptive 
motion cues. “Mental workload” is the operator’ s evaluation of his attentional-load margin while 
performing adequately in a specified task context. 

A review is given of a wide range of simulations in which operator steering control of a vehicle 
is involved and the: dominant-cues, closed-loop bandwidth, measured operator effective time-delay, 
and ratio of bandwidth-to-inverse delay, are summarized. For well designed and low mental work- 
load conditions, this ratio is well under 1.0, as control theory would dictate. However, for important 
human tasks like fast locomotion, the ratio approaches 1.0 (limited by the inverse delay limit of 
about 8 rad/sec), and the mental workload becomes critical. These imply requirements that the eye- 
ball stabilization and pursuit subsystems have higher bandwidth. More importantly, it dictates that 
any simulation delay artifact must be less than about 1/4 of the effective operator delay in a given 
task, i.e., less than about .04 seconds delay artifact for locomotion simulation. 

A correlation of kinetosis (simulator-sickness) with dynamic scene field-of-view (FOV) is 
shown. It has been found that, to avoid kinetosis, the FOV should span less lateral view than a curve 
of about 1 . 10 Gy would cover at real-world speeds (typically, less than 60 degrees). Other require- 
ments include: framing of scene “streamers” and avoidance of perceivable image jitter or “strobing.” 

The visual resolution of a virtual environment should allow reading a newspaper subheadlines at 
arms length. High Definition Television (HDTV) would pass this test. 

The use of moving base simulators to improve the validity of locomotion teleoperations is dis- 
cussed, and data are shown to support the following recommendations. Linear motion bases are not 
needed if the real-world task entails less than about 1. 1 Gy of transverse acceleration. If the rms Gy 
is greater, it is more important to maintain correct motion-cue phasing (<30 degree disparity) than 
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amplitude, which can be cut to about 1/5 the actual levels. Rotary motion rates are sensed and used 
as “rotary motion dampers” when the real-world task is regulation against inertial motion distur- 
bances (as in running), but are ignored when the task involves pursuit of a maneuvering target. 
Details are discussed in the presentation. 

Some rules-of-thumb for good “feel-system” simulation, such as for control manipulanda are 
given. Their vernacular version dictates that “good” simulation of a: 

- zero inertia and force-free control should “feel like a stick of balsa wood” (negligible 
friction, jerk or jitter) 

- hard stop should “feel like a brick wall” (no creep or sponginess) 

- coulomb friction should “feel like sliding a refrigerator magnet” (remain in place; no creep, 
bounce or jitter) 

- centering detent should “yield an audible “klunk” when traversed” (no lag or sponginess) 


Try these on your favorite manipulandum! 

Finally, simulation tests of teleoperators and virtual environments should include three types of 
measures: system performance, operator (or robot) “behavior” (e.g., describing functions, etc.), and 
mental workload evaluations. The criteria for adequate mental workload measures are that they be: 


1. Relevant 

2. Sensitive 


3. Concordant 

4. Reliable 

5. Convenient 


to mental workload, per se 
monotonic correlation with workload 
insensitive to irrelevant variables 
high test power (covariance/residual) 
ubiquitous trends in target population 
proven test: retest repeatability 
proven norms and statistics 
easy to administer and learn 
portable 

low cost for adequate level of reliable result 


Properly designed subjective ratings usually fare better than such measures as evoked response 
potentials! Examples of the foregoing approach are given, in optimizing motion washouts. 


If your teleoperator or virtual environment simulation can pass these rule-of-thumb tests, then 
you can be fairly sure that it will provide results which can stand the final test of real-world valida- 
tion, form a proper data base for scientific modeling and engineering design, and be liked by its 
users. 
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CONTROL BW AND H.Q. DELAYS ARE RELATED 


SIMULATION 
; Example) 

DOMINANT CUES 
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IMPLICATIONS - VISUAL DELAYS 

• Optimized control tasks have BW/t" 1 « 1.0 

• Human will adapt to * 1/5 BW, down to t « .20 sec 

• For high BW tasks: BW/t" 1 * 1.0 and T m - n > .16 sec 

• Control theory shows that any simulation delay artifacts must be «J2Q 
• sec to provide “valid" simulation 

Criterion 1 

• a "Valid" simula tion requires delay artifacts be « .20 sec: 

<.02 sec (60Hz) = Excellent; <.04 sec = Good; >.10 sec = Invalid 
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FIELD OF VIEW VS. KINETOSIS 

Fixed-Base Vertigo vs. Reid of View 


Probability of Vertigo ( Short -Term "Awo rnt ts") 



implications: For wide FOV use "impoverished scene" 

Criterion 2: Keep FOV to cover less lateral view than a curve of ±.10 Gy would 
cover at real-world speeds y 

Other factors: 

• More parafoveal details worse conflict 

• Proper framing of parafoveal streamers is important 


VISUAL UPDATE AND RESOLUTION OF DISPLAYS 


Criterion 

• For "inner loop" control of path rate and curvature, "streamers" from 
scene microtexture should not jerk or strobe backward. 

(* 20-60 frames/sec update) 


Criterion 

• For "outer loop" guidance and navigation must be able to perceive and 
understand cues at = 3-5 sec ahead of vehicle 
Implies = 2 min-arc of resolution for signs, potholes, targets (e.g„ HDTV) 


Criterion 

• For Virtual Environment should be able to read a newspaper at arm’s 
length. (Subheadlines = OK) (e.g., HDTV) 
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VERIFICATION OF VALID SIMULATION 
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MAN-MACHINE INTERFACES IN HEALTH CARE 


Steve Charles, M.D. and Roy E. Williams 
Center for Engineering Applications 
Memphis, Tennessee 

The surgeon, like the pilot, is confronted with an ever increasing volume of voice, data and 
image input. Simultaneously, the surgeon must control a rapidly growing number of devices to 
deliver care to the patient. The broad disciplines of man-machine interface design, systems integra- 
tion and teleoperation will play a role in the operating room of the future. The purpose of this com- 
munication is to report the incorporation of these design concepts into new surgical and laser deliv- 
ery systems. A review of each general problem area and the systems under development to solve the 
problems follows. 



Ophthalmology is a machine-dependent specialty embracing the patient-machine interface and 
the surgeon-machine interface. As many as fifteen independent systems may be used in the operating 
room during a surgical procedure. Each system is currently manually controlled creating an intensive 
operator interface environment. The surgical operating system known as the Ocular Connection 
Machine (OCM) has reduced the number of stopcocks, foot pedals, adapters, displays, controls and 
power cords by several orders of magnitude. Physiological analog parameters, including intraocular 
temperature and pressure, previously monitored with human interaction are now controlled by servo 
systems, while surgical analog parameters, i.e„ cutting and thermal, are controlled by dual propor- 
tional positioning of the surgeon’s foot. Multiple displays have been replaced by one thus reducing 
viewing time and errors. By reducing the amount of man-machine interactions, the system markedly 
reduces set-up time and clean-up time, as well as improving reliability, functionality and patient 
safety. 

Laser treatment of the retina is an intensive man-machine task. The ophthalmologist must 
determine the area of treatment, aim the beam and fire while the patient is awake. The Laser Imaging 
Work Station (LIWS) currently under development alleviates much of the ophthalmologist-machine 
interaction and reduces the discomfort of the patient-machine interaction. The LIWS is an integrated 
system containing a high-speed digital correlation retinal tracker, an automated laser delivery system 
which controls laser power, wavelength, spot size, duration and location, and an expert system with 
algorithms to avoid the macula, optic nerve, vessels and previous photocoagulated areas. Surgeon 
interface occurs off-line whereby a patient treatment template is determined and down-loaded to the 
laser delivery system. 


Real-time diagnostic imaging of a patient’s eye is currently resolution limited, two-dimensional 
and user interaction intensive. A system is currently under development which provides quasi-real 
time stereo (3D) ultrasound and improved lateral resolution. Depth of the viewing volume within the 
eye (focal plane) is identical to current focus mechanisms used by ophthalmologists whereby a hand- 
held lens is moved forward and backward yielding different viewing planes (indirect 
ophthalmoscopy). This hand-eye coordination and resulting spatial information will be natural for 
ophthalmologists. 
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Utilization of advanced robotic systems in a teleoperation environment will enable microsur- 
geons to improve present operations and allow those surgeons not as dexterous to perform operations 
not manageable with present systems. For example, epiretinal membranes too thin to remove without 
risk of macular holes would be a realizable target for the ophthalmic microsurgeon. By applying 
6 degrees-of- freedom in a coarse-fine strategy with translational resolutions of 10 to 20 microns, 
these techniques and others are made possible. The surgeon controls the bilateral robotic system 
through the surgeon-machine interface (SMI) which provides stiffness feedback by sensing force and 
position at the operating site and backdriving actuators within the SMI. Further sensing of optical, 
electrical and mechanical impedance at the operation site is fed back to the surgeon via the foot 
pedal interface, offering another realm of sensory feedback. 
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TELEROBOTICS IN REHABILITATION: BARRIERS TO A VIRTUAL 

EXISTENCE 





Larry Leifer, Machiel Van Der Loos, and Stefan Michalowski 
Stanford Center for Design Research 
Stanford, California 


The Need for Telerobotics In Rehabilitation 

There are over 67,500 quadriplegics in the United States today, with an estimated 2,400 to 4,000 
new injuries occurring each year (Stover et al., 1986). These injuries occur most frequently among 
young males between the ages of 16 and 30. Due to advances in medical treatment (antibiotics and 
skin care), these individuals now have a relatively normal life expectancy. They are alive, but dra- 
matically cut off from most aspects of a normal existence. They live, virtually, through the actions of 
others. Perhaps, life would be more fulfilling if they were also offered the opportunity to directly 
control their environment through telerobotic tools for independent living. Virtual telerobotic envi- 
ronments for the rest of us could bring reality to these individuals. 


It is estimated that caring for a quadriplegic, including standard medical treatment, equipment 
maintenance and attendant care, costs about $47,000 (1980 dollars) per year. This translates to 
approximately 1.9 million dollars over forty years (Kalsbeek et al., 1980). The net direct cost to the 
Department of Veterans Affairs will be approximately $5 billion for its current population of service 
connected quadriplegics. On the promising side. Young et al (1982) estimated that every dollar spent 
for rehabilitation research and development returns $1 1 in cost benefits to society. In the case of 
quadriplegia, the cost of attendant care (nominally $25/hr to the insurer and increasing) can be 
reduced by providing personal telerobots (nominally $5/hr and decreasing). Hammel et al. (1989) 
demonstrated that telerobots can satisfy the vocational manipulation needs of personal computer 
users for periods of over four hours at a time without attendant intervention. Employment makes it 
possible to recover some or all of the indirect cost of severe disability. 

Barriers to Telerobotics Technology in Rehabilitation and Health Care 

If the direct cost of severe disability is so high, and telerobotics technology is available to help 
reduce costs, then what have been the barriers to its widespread acceptance and deployment? Clini- 
cal experience with telerobots suggests that there are several key barriers: 

Social Barriers: As a society, we place little emphasis on restoration of function for persons 
with disability, we prefer to “take care of them”. Because the economics of cost and ben- 
efit are not coupled, we fail to see the opportunity. However, even if we began deploy- 
ment today, our society has educated too few persons to support the advanced assistive 
devices (i.e., we have enough researchers to create independent living tools, but too few 
development and service persons to support clinical and domestic usage). These are the 
dominant factors impeding wider adoption of advanced technical aids for persons with 
disability. 
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Institutional Barriers: Government programs are scattered and disjoint. There is no system- 
atic vision and considerable inter-agency protectionism. Perhaps most significantly, the 
“conflict of interest” witch hunt in government makes it virtually impossible to transfer 
laboratory results to the commercial sector in a timely and cost-effective manner. For its 
part, the commercial sector does not take the investment risk to develop these devices 
because it is not clear that third party reimbursement will be forthcoming. 

Technical Barriers: Here, at least, the science and engineering communities do have some 
control of the issues, these include: 

The human-machine-interface is the dominant technical barrier to widespread use 
of telerobots in rehabilitation. Text, voice, graphic and kinematic command-con- 
trol interfaces are very cumbersome for robot motion specification, planning and 
supervision. This forces the user to be overly dependent on pre-programmed 
motions and the technicians who create them. We must work towards 
“instructable” telerobots. 

The machine-environment-interface is the second most deficient aspect of teler- 
obotics in rehabilitation. The absence of sensor driven grasp and object approach- 
avoidance reflexes forces the user to directly control end-effector motion under 
difficult observational circumstances and without “natural kinesthetic cues”. We 
must develop robust sensate end-effectors and the “reflexes” to make them useful. 
Force (impedance) control will be a requirement for advanced user support tasks. 

Mobility, or the lack of it, defines the telerobot’s work space and, in part, its ultimate 
utility. One can not reasonably expect general cost-effectiveness when people 
must be available to bring work to the telerobot. Raw mobility is, however, not 
enough. Remote presence makes much greater demands on the user-interface and 
telerobotic sensory capability. A more “intelligent” robot may, in fact, be the 
greatest challenge yet to the user-interface designer. 

Fault-intolerance is an overriding shortcoming of almost all robots. As program- 
mable electro-mechanical systems, they are inherently subject to a very wide 
variety of fault modes. Not even the digital controller in current machines take 
advantage of computer fault-tolerance architectures (which themselves make no 
provision for sensor and actuator failure modes). Widespread personal use of 
telerobots will require fundamental progress in design for fault-tolerance (we must 
get well beyond just being careful). 
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Telerobotics in Rehabilitation: Once Around the World 


A small number of underfinanced telerobotics teams around the world are attempting to over- 
come these barriers. The most recent compilation of papers are in the Proceedings of the “First Inter- 
national Workshop on Robotic Applications in Medicine and Health Care” (1988). The next edition 
will grow out of the International Conference on Rehabilitation Robotics (sponsored by the A. I. 
duPont Institute, June, 1990). The following synopses highlight some of the ongoing R&D (see 
Table-4 for a technical comparison of the telerobots used. 

The Johns Hopkins University Applied Physics Laboratory (JHU/APL) project (Schneider et al., 
1981) concentrated on the implementation of a workbench-mounted robot intended to perform 
activities of daily living (ADL) tasks. The design (initially 4 degrees-of-freedom then extended to 5) 
was derived from prosthetic arm technology. Control is accomplished by a head-motion (chin) con- 
trolled joystick for joint specific motion. Pre-programmed motions are invoked by menu-selection 
and input command via a sip-and-puff switch. 

The Tufts-New England Medical Center robotics project (Gilbert and Foulds, 1987; now at the 
A I duPont Institute) concentrated on the design of a universal robot programming language, 
CALVIN, to provide a common interface to the many different manipulators available. Using 
CALVIN, they set up a variety of small, low cost, robot work cells (typically 4 and 5 degree-of- 
freedom manipulators) in clinical rehabilitation settings for disabled children. The clinical objective 
is to foster intellectual development of the child. 

The Boeing Company developed a voice-controlled workstation using the Universal Machine 
Intelligence RTX (5 degree of freedom) manipulator (Fu, 1986). The distinctive feature of this pro- 
ject is that it’s user interface is a voice query system for large data bases developed by Boeing. Prab 
Command, Inc., began marketing the system in 1988 for vocational applications. There is very little 
telerobotic function beyond pre-programmed diskette loading. The internal project at Boeing has 
ended and Prab sold the technology to the Zenith-Heath. 

From 1980 through 1988, the Veterans Administration sponsored a collaborative effort with 
Stanford University (Leifer, 1982; Hammel et al., 1989) to test the hypothesis that industrial manipu- 
lators could cost effectively serve the needs of severely impaired spinal cord injury patients. The 
project demonstrated that, through voiced commands, the utility and reliability of a high perfor- 
mance manipulator (PUMA-260) and control language (VAL-2) can yield attractive cost-to-benefit 
performance ratios when the telerobot is able to operate for four hours without attendant interven- 
tion. Sensate end-effector, mobility, 3D point designation and natural language studies laid the foun- 
dation for further R&D. A desktop vocational assistant robot (DeVAR) version of the system is 
available commercially . 

Outside the United States, the Canadian Neil Squire Foundation has developed a low-cost 
manipulator designed for desktop applications in rehabilitation (Cameron, 1986). The system is 
being sold by the foundation. At the Institute for Rehabilitation Research in the Netherlands, Hok 
Kwee (previously with the French Spartacus project (Kwee, 1983) and his colleagues have devel- 
oped a wheelchair mounted, joystick-controlled, manipulator (MANUS, Kwee, 1987). They are 
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particularly interested in collocation of the robot and user at all times. The manipulator is expected to 
function as an arm, not really a telerobot. 

While human attendant care is the norm today, and telerobotics technology promises to release 
some attendant time cost effectively, there is a third alternative. Capuchin monkeys have been 
trained to provide attendant services from feeding to appliance operation (internal VA program 
review, 1988). Monkey training is expensive, labile, and done by methods objectionable to the ani- 
mal rights community. Program retention (to a maximum of 12 short tasks) and willfulness remain 
significant obstacles to the monkey assistant concept. The approach is particularly limited by the fact 
that monkeys can only work one-on-one with their user. The distractions of institutional and voca- 
tional settings render the monkey ineffective. 

Table 1 presents an extended comparison of the strengths, capabilities and limitations of five 
approaches to augmenting the independence of severely impaired persons. Supported by a growing 
body of experimental evidence, this comparison strongly suggests that telerobots can become cost- 
effective personal and vocational assistants. Table 2 lists many of the feeding, personal hygiene, 
vocational, and recreational tasks that have been demonstrated over the past 8 years by four genera- 
tions of the Stanford- VA Rehabilitation R&D Center’s Telerobotic systems. Table 3 identifies the 
technical functions required to perform these tasks. 

A Partial View of the Future 

The operator interface will remain the dominant problem. It is so difficult that most systems 
designers prefer to ignore it and focus on “tangible” technical specifications. It is likely to use mul- 
tiple-channels in the sense of incorporating a “natural robot instructional language” and bi-direc- 
tional pictographic dialog between the operator and the system. Increasingly “autonomous” teler- 
obots will actually increase the burden on this interface. 

Sensor driven motion planning and autonomous grasp/avoidance reflexes will become common- 
place. The rate of introduction of such features will, however, be much slower than expected. In part, 
this is due to the fact that system architectures will continue to be “fault-intolerant” such that the 
introduction of both sensors and programmed reflexes will bring new reliability problems with them. 

Force (impedance) control of telerobots will continue to evolve slowly even thought this capa- 
bility is a fundamental requirement for any robot that must work intimately with humans. Physical 
therapy by robots is both needed and impossible without force control. These lines of technical evo- 
lution will themselves depend on getting more applications feedback from telerobots in the clinic, 
home and office. In combination, we see a rather daunting challenge. 
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Table la In-depth Review of the Capabilities, Strengths and Weaknesses of Five Basic Approaches 
to the Restoration of Manipulative Function for People with Severe Physical Limitations (none = 
limited = *, usable = **, good = ***, excellent = ****). 


Environment Controllers 


Human Input/Output Factors 


COMMAND * 

Substitution errors 5% 

Detection errors 10% 

Number of commands 10 


Syntax options 
User programming 
CONTROL 
Real-time 

Degrees of freedom 
Flexible 

User programmable 

DIALOG * 

Feedback modes lights 

Explanations 
Inference 
Rule based 
User adaptive 
Model based 
Machine Input/Output 

AUTONOMOUS PLANNING 
Path 
Strategy 
Data driven 

PROGRAMMABLE REFLEXES 
Force compliance 
Contour following 
Proximity sensing 
Collision avoidance 
POSmON/ORffiNTATION 
Degrees of freedom 
Radius of working volume 
Precision 
Repeatability 
Strength 
Speed 

MOBILITY 

Degrees of freedom 

Range rooms 

Remote control IR link 

Autonomous 
GRASP 

Degrees of freedom 
Grip force 
Dexterity 
SENSATION 
Tactile 
Force 
Proximity 
Vision 
Audition 


Prosthetics Workstations - Seamone 
Monkeys - Willard 

Robots - Leifer 

People 

** 

* 

*** 

**** 

5% 

10% 

10% 

1% 

10% 

25% 

15% 

2% 

20 

12 

100 

2000 

menu 

menu 

program 

English 

- 

- 

yes 

yes 



*** 

- 

- 

- 

yes 

- 

- 

- 

7 

- 

- 

- 

yes 

- 

- 

- 

yes 

- 

* 

* 

*** 

**** 

lights 

noises * 

voice 

voice 


- 

status 

unlimited 

- 

- 

limited 

unlimited 

- 

- 

limited 

general 

unlimited 

- 

? 

limited 

- 

? 

sensory 

general 


. 

** 

**** 

- 

- 

limited 

general 

- 

- 

limited 

general 

- 

- 

limited 

general 

- 

* 

** 

**** 

_ 

** 

** 

*** 

_ 

.. 

** 

**** 

_ 



. 

_ 

*** 

* 

**** 

* 

** 

*** 

**** 

4 

9 

7 

9 

40cm 

30cm 

40cm 

55cm 

low 

low 

high 

flexible 

2mm 

3mm 

0.2mm 

3mm 

low 

low 

low 

flexible 

low 

moderate 

flexible 

flexible 

- 

*** 

** 

**** 

_ 

6 

4 

6 

desk 

room 

rooms 

unlimited 

- 

voice 

voice 

voice 

- 

? 

limited 

unlimited 

* 

*** 

** 

**** 

1 

6 

1 

6 

1kg 

1.5kg 

3kg 

25kg 

minimal 

good 

minimal 

excellent 

- 

**** 

** 

**** 

_ 

**** 

* 




** 

**** 



** 


_ 

**** 

_ 

**** 

_ 


_ 

**** 
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Table lb 


Assessment 

PERFORMANCE 
Task time 
Training time 
Number of tasks 
Commands per task ' 
Precision 
Repeatability 
Reliability 
Accessibility 
SAFETY 

Intention errors 
Intrusion errors 
Contact errors 
COST 

Hardware 
Software 
Training 
Maintenance 
Cost/hour of use 


Environment Controllers 

Prosthetics Workstations - Seamone 




Monkeys 

-Willard 





Robots - 

Leifer 





People 

** 

** 

** 



seconds 

minutes 

minutes 

minutes 

seconds 

2hrs 

30hrs 

20hrs 

40hrs 

4hrs 

10-20 

15-30 

10-20 

20-200 

40-400 

1 

1-4 

1-4 

1-20 

1-2 

- 

5mm 

2mm 

0.2mm 

2mm 

- 

1mm 

3mm 

0.2mm 

3mm 

high 

good 

fair 

good ' 

fair 

24hrs 

24hrs 

12hrs 

24hrs 

8hrs 

**** 

*** 

** 

** 


5% 

10% 

5% 

10% 

4% 

0% 

4% 

5% 

5% 

2% 

0% 

2% 

4% 

2% 

1% 

***# 

** 

** 

** 

* 

low 

medium 

medium 

high 

low 

low 

low 

high 

high 

low 

low 

medium high 

medium 

low 

low 

medium 

medium 

low 

high 

$I/hr 

$4/hr 

$10/hr 

$5/hr 

$25/hr 
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Table 2 Meal preparation and service, vocational material handling, personal hygiene, and recre- 
ational tasks that have been performed for and with disabled individuals across four generations of 
desktop robotic assistants (DeVARs I, II, M, IV) 


Meal preparation and service 

Vocational material handling 

Personal hygiene 

arrange table setting 

write with pen and/or pencil 

wash and dry face 

open and close microwave door 

retrieve books and manuals 

brush teeth 

open and close refrigerator door 

set up books for reading 

dispense toothpaste 

manipulate containers 

turn book and report pages 

use electric shaver 

set appliance timer 

turn on-off computer equipment 

retrieve mouthstick 

pour liquids and solids 

type on keyboard 

comb and brush hair 

beat eggs 

adjust keyboard position 

blow nose 

toss salad 

operate private and speaker phones 

apply make up 

soup preparation and service 

insert and retrieve diskettes 


heat and serve casseroles 

insert and retrieve audio tapes 


serve pudding 

operate dictation equipment 


serve fruit 

manipulate printout 


prepare and serve spaghetti 

voiced control of generic software 


prepare and bake a cake 

load and operate printer 


use knives, forks and spoons 

retrieve and serve medication 


retrieve drinks 

circuit boards for inspection 


mix drinks 

operate electronic control units: 


lock and unlock cabinet doors 

door operation 


light and extinguish candles 

security system 


light and extinguish cigarette 
open and close storage drawers 

stereo equipment 


room lights 

Recreation 


window open and close 

paint and sketch 

arrange flowers 

hand out flowers 

hand out candy and souvenirs 

perform one armed ballet 

checkers 



monopoly 



television 

operate video games 



pac-man 

invaders 

play board games: 
chess 
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Table 3. Several functional capabilities are needed in a Telerobotic Assistant to achieve utility in the 
performance of independent living tasks, not needed ( ), should have (□), must have (■) 


MAINTENANCE Tasks 

Food Preparation 
Food Service 
Personal Hygiene 
Personal Grooming 
Clothing Management 
Appliance Usage 
VOCATION Tasks 
Storage & Retrieval 
Equipment Operation 
Assembly 
Word Processing 
Computing 
Materials Processing 
RECREATION Tasks 
Reading 
Film & Video 
Performing Arts 
Graphic Arts 
Sports 
Social Interaction 


COMMAND 


CONTROL 

DIALOGUE 


PLANNING 

GRASP 


MANIPULATION 

Ifieflexes 

ioBILITY 
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INTRODUCTION 


Computing machines have brought new opportunities as well as challenges to our information 
age. While computer capacities and speeds have increased remarkably, our ability to communicate 
with these information engines is still limited by poor interfaces between the human and the com- 
puter. To harness and exploit the power of the computer as a tool and extender of our intellect, it is 
essential that we build new interfaces which are intuitive, transparent and take advantage of our 
innate spatial abilities. 

The new technology of virtual interfaces provides a revolutionary way to improve the coupling 
of the human to the computing machine. Virtual controls and displays allow panoramic presentations 
in three dimensions to be made to the eyes, ears and hands of the user by projecting “virtual” images 
into the senses. To gain this global experience “the human literally wears the computer.” Virtual 
interfaces transfer stimuli to the eyes, ears and hands, and use head, eye, hand movement and speech 
to control the machine. In this way the operator can be surrounded by a “circumambience” of com- 
puter synthesized information which becomes a spatial world through which he and machine can 
effectively communicate. The operator, in turn, interacts with his global medium by looking at 
objects, pointing his hands and giving verbal commands. The medium also allows virtual objects, 
which appear to be real but which are virtual projections, to be touched and manipulated by the 
hands. In order to create the virtual space representation of information, the virtual display hardware 
components are programmed with “mindware.” Mindware is a special class of software that takes 
into account the perceptual organization of the human and dynamically creates three-dimensional 
sound, video and tactile images which appear as an artificial reality surrounding the user. 

The virtual interfaces described above not only solve many existing interface problems but 
empower new and novel interfaces for operators in aerospace vehicles and other applications such as 
teleoperation, computer-aided design, medical imaging, prostheses for the handicapped and enter- 
tainment. Even though some aspects of virtual interface technology have been under development 
for two decades, there are still several key problems which must be addressed and resolved to make 
these concepts practical for research, industrial and consumer applications. These problems are 
summarized below: 
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THE PROBLEMS 


1 . We don’t understand the human factors of virtual space, 

2. We don’t know how to measure the “goodness” of virtual worlds, 

3. Except for the more expensive military versions, virtual displays lack sufficient resolution for 
wide field-of-view presentations. 

4. Position sensing technology (for control and image stabilization) needs improvement to 
increase precision and update rate while reducing noise and sensitivity to conductive materials. 

5. Current graphics engines have limited update rate and polygon count. 

6. The headgear is too heavy. 

7. There are too many wires. 

8. There is a need for a universal software architecture and tool kit for supporting future devel- 
opment and application of virtual worlds, 

9. Virtual interfaces are too expensive. 

10. There is no forum currently for discussing the development and applications of virtual worlds. 

POSSIBLE SOLUTIONS 


As research and development laboratories in universities and industry respond to the growing 
excitement about virtual worlds, we would propose that a research agenda be established to solve the 
problems above. Here are recommended approaches: 

1 . Establish a national knowledge base regarding the ergonomics, technology and application 
of virtual interfaces. Build a centralized literature collection, and interface with the Crew System 
Ergonomics Information Center (CSERIAC) at Wright-Patterson AFB, OH. Add to the Handbook of 
Perception and Human Performance and the Engineering Data Compendium published by the Air 
Force. Begin a scholarly journal to address issues of virtual interfaces including the psychophysics, 
technologies and applications of virtual worlds. 

2. Explore new psychophysical and neurophysiological measurement techniques which operate 
within virtual worlds. 

3. Develop high resolution parallel image projection approaches which reduce bandwidth and 
don’t require the production of an aerial image. 
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4. Develop position measurement approaches which are passive. 

5. Improve image stabilization (close loop outside graphics processor). Develop graphics pro- 
cessor to support parallel information input. 

6. Eliminate wires with passive sensing and data links. 

7 Create microscanner for Maxwellian view image projection. 

8. Create mindware to take into account the perceptual organization of the human. Standardize 
on network protocols for tele virtuality and transportability of software and applications. 

9. Establish a special interest group on virtual interface technologies under the aegis of the 
ACM, IEEE or HFS. 


CONCLUSIONS 


Virtual interfaces provide a bold new opportunity for solving many of the perplexing problems 
of interfacing human and machine intelligences. As pioneers, we are obligated to pursue die devel- 
opment of virtual interface technologies in a systematic way and to leave a technology base and tools 
as our legacy for others to build upon. If developed systematically, virtual interfaces can be one of 
the greatest advances of our age. 
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f INTERACTING WITH VIRTUAL WORLDS 







David Zeltzer 

Massachusetts Institute of Technology 
Cambridge, Massachusetts 


What tools should we provide the user for defining and interacting with objects and agents in 
virtual environments at varying levels of complexity? Understanding the appropriate simplifications 
to make is critical for modeling non-trivial environments and agents with varying levels of auton- 
omy. I describe a set of abstraction mechanisms appropriate for constructing and interacting with vir- 
tual worlds, and I discuss how programming, and direct manipulation or guiding techniques, can be 
used to afford users interactive access to graphical simulations at appropriate levels of abstraction. 

Structural abstractions provide a mechanism for describing the kinematic structure of objects, 
such as transformation hierarchy for a jointed figure, and for defining physical attributes of objects, 
e.g., mass, stiffness, etc. 

Procedural abstractions are a mechanism for defining processes that control rigid and non-rigid 
object motion, such as elastic deformation, collision detection, forward dynamics or inverse kinemat- 
ics, independent of the structure of the agen or object 

Functional abstractions are the means with which we associate procedures with objects and 
object sub-assemblies, for the purpose of defining meaningful behaviors. This allows complex, 
largely unmanageable systems with many DOFs - like the human figure, with over 200 DOFs — to be 
decomposed into a set of small, manageable subsystems, each with few DOFs. 

An agent consists of a structural definition, a set of functional units which define the behavior 
repertoire of that entity, and some means for selecting and sequencing (possibly concurrent) behav- 
iors. The mechanism for selecting and sequencing these motor skills must link action with percep- 
tion. 

The lowest layer of abstraction is represented by objects and structures with no procedural 
attachment, and is called the machine level. At the other extreme, the task level, guiding and pro- 
gramming tools interact with agents through their repertoire of behaviors. Task level interaction 
emerges when guiding and programming techniques are applied at higher layers of abstraction. 

Guiding and programming tools have a dual nature: they can be used to construct and modify 
instances of any of the abstraction types; or they can be used at runtime as control instruments. I 
describe how programming and guiding can be used with each of the four classes of abstraction. 
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1. INTRODUCTION 

The utility of virtual reality computer graphics in telepresence applications is not hard to grasp and 
promises to be great. When the virtual world is entirely synthetic, as opposed to real but remote, the 
utlility is harder to establish. Vehicle simulators for aircraft, vessels, and motor vehicles are proving 
their worth every day. Entertainment applications such as Disney world’s StarTours are technologi- 
cally elegant, good fun, and economically viable. Nevertheless, some of us have no real desire to 
spend our lifework serving the entertainment craze of our sick culture; we want to see this exciting 
technology put to work in medicine and science. 

2. TESTING A FORCE DISPLAY FOR SCIENTIFIC VISUALIZATION: MOLECULAR 
DOCKING (Ming Ouh-Young) 

2.1 The technology. This test uses a Model E-3 Argonne Remote Manipulator (the ARM) 
that offers a full 6-D input position sensing and full 6-D force and torque output. A Sun 4 provides 
computational power and ARM control; and E.&S. PS-300 provides the visual output. A Tektronix 
stereo plate is used with passive circular polarizing glasses. 

2.2 The molecular docking application. The enzyme dihydrofolate reductase binds to a 
variety of inhibitors, natural and synthetic. Of twelve inhibitors studied by our collaborators at 
Burroughs-Wellcome, the precise structure of the enzyme-inhibitor complex is known for six. 

Binding energy is a complex function of atomic positions, involving Coulomb forces, van der Waals 
forces, nuclear forces, hydrogen-bonding forces, and thermodynamic forces. With 6 degrees of free- 
dom in the drug docking, and up to 12 twistable bonds in the drugs, the docker is attempting to find a 
global minimum of the energy in a space of up to 18 dimensions, a space pock-marked with local 
minima. Brute-force algorithmic methods would require years of computer time to explore the space. 

It is hard to imagine a visual display that would effectively produce all the needed insight to enable 
a user to manually dock the drug so as to minimize energy. We postulated that feeling the forces, 
particularly the (l/r)**7 and (l/r)**13 forces that define the receptor pocket, would substantially 
help chemists in manual docking. 

2.3 The experimental results. Chemists can dock an inhibitor in about 25 minutes using 
our force display system. Ming’s controlled experiments indicate an upper bound of about a factor of 
two improvement in docking performance when the visual display is aided with our force display. 
Our force display has many primitive characteristics, so there is hope for some improvement in this 
number, but an order-of-magnitude improvement doesn’t seem to be in the cards. 
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2.4 Whither now? We plan to get some actual drug-design experience with real drug 
designers. We also want to build or buy a finger-scale, versus arm-scale, 6-D force output device. 

The literature suggests that resolution should be at least as good with the scaled-down device, and 
stability and ease of use much better. 

3. TESTING A HEAD-MOUNTED DISPLAY FOR SCIENTIFIC AND MEDICAL 
VISUALIZATION (James Chung) 

3.1 The technology. We use the standard available technology: tiny TVs and Polhemus 
trackers. Our PixelPlanes graphics engine is faster than almost all of today’s delivered engines. The 
Polhemus has respectable frame-rate but unacceptable lag for head-mounted displays. We measure 
250-400 msec., and lag seems to depend upon source-sensor distance. We are pursuing several new 
tracker technologies. 

3.2 The only advantage of head-mounted displays - Intuitive navigation. Considered 
cold-bloodedly, today’s head-mounted displays have many disadvantages: resolution, sharpness, 
field-of-view. Their inherent comparative advantage is intuitive navigation of the virtual space. It 
should be much easier for the user to know where he is and what he is looking at than in through-the- 
window graphics. How much is this worth? How much does it improve performance in a real 
application? 

3.3 The radiation-treatment planning application. Radiation oncologists have to plan the 
shape, intensity, and direction of multiple beams of radiation so as to bum the tumor without burning 
other sensitive or vital organs. This task requires imaginative use of 3-space, a good image of the 
anatomy of the patient, and, ideally, real-time calculation of the dose to the tumor and to unwanted 
organs. 

3.4 The planned tests. We have volume-visualized MRI anatomical data on tumor patients. 
Our collaborating radiologists, in a controlled experiment, will devise treatment plans using the 
head-mounted display versus through-the-window graphics. We will measure task time and 
goodness of plan as experimental variables. 
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USING VIRTUAL REALITY AS A TOOL TO STUDY THE RELATIONSHIP 
BETWEEN HUMAN BEINGS AND PHYSICAL REALITY 

J. Lanier 

VPL Research, Inc. 

Redwood City, California 


This presentation focused on the following areas: 

1. VPL’s Virtual Reality system (the RB2); What it is and what has been learned from it. 

2. Unsolved technical problems in Virtual Reality. 

3. Cultural, philosophical, and political inquiries demanded by. Virtual Reality. 

4. The upcoming Reality Net project. 
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THE AMES VIRTUAL ENVIRONMENT WORKSTATION: 
IMPLEMENTATION ISSUES AND REQUIREMENTS 


\ 



S. Fisher, R. Jacoby, S. Bryson, P. Stone, I. McDowall, M. Bolas, D. Dasaro, E. Wenzel, 

C. Coler, and D. Kerr 
NASA Ames Research Center 
Moffett Field, California 


This presentation describes recent developments in die implementation of a Virtual Environment 
Workstation in the Aerospace Human Factors Research Division of NASA’s Ames Research Center. 
Introductory discussions are presented on the primary research objectives and applications of the 
system and on the system’s current hardware and software configuration. Principal attention is then 
focused on unique issues and problems encountered in the workstation’s development with emphasis 
on its ability to meet original design specifications for computational and graphics performance and 
for associated human factors requirements necessary to provide compelling sense of presence and 
efficient interaction in the virtual environment. 

1. Research Objectives and Applications 

In the Aerospace Human Factors Research Division of NASA’s Ames Research Center, an inter- 
active Virtual Environment Workstation (VIEW) has been developed to aid in design, simulation and 
evaluation of advanced data display and management concepts for operator interface design. The 
VIEW system provides a virtual auditory and stereoscopic image surround that is responsive to 
inputs from the operator’s position, voice and gestures. As a low-cost, multipurpose personal simu- 
lation device, this variable interface configuration allows an operator to virtually explore a 
360-degree synthesized or remotely sensed environment and viscerally interact with its components. 

At present, the major thrust of this research effort centers on the development of guidelines and 
requirements that are necessary for implementation of virtual environments that are matched as 
closely as possible to human cognitive and sensory capabilities and that feature transparent, natural 
system interaction. It is projected that design specifications derived from this prototype system can 
help define the range of necessary and sufficient hardware and software architectures for future vir- 
tual environment systems. 

Application areas of the virtual environment research are focused in: 

TELEPRESENCE: Development of workstations for complex operational tasks such as teler- 
obotic and telepresence control of remotely operated robotic devices and vehicles that require a suf- 
ficient quantity and quality of sensory feedback to approximate actural presence at the task site, 

DAT ASP ACE: Development of concepts and guidelines for “portable” multi-modal information 
management systems such as EVA Spacesuit visor display, with subsequent development of work- 
stations for supervision and management of large-scale integrated information systems in which data 
manipulation, storage and retrieval, and system monitoring tasks can be spatially organized. 
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VISUALIZATION: Design and evaluation of virtual environments to facilitate visualization of 
complex, three-dimensional data structures in areas such as architecture, engineering and computa- 
tional fluid dynamics, and for interactive training or planning environments such as surgical 
simulation. 

2. VIEW System Configuration: Performance requirements and implementation 

The current Virtual Interface Environment Workstation consists of: a wide-angle stereoscopic 
display unit, glove-like devices for multiple degree-of-ffeedom tactile input, connected speech 
recognition technology, gesture tracking devices, 3D auditory display and speech-synthesis technol- 
ogy, and real-time computer graphic and video image generation equipment. This hardware is inte- 
grated with a realtime Unix workstation that supports the computations required to drive an external 
high-performance realtime 3D graphics system, processes input from up to 12 realtime input periph- 
erals (e.g., the trackers and gloves), and provides other realtime task processing. A collection of 
software called the “simulation framework” has also been developed that consists of a well-docu- 
mented library of functions to provide access to all of the system peripherals and software services, 
and of a collection of source files and simulation software that demonstrates the use of the major 
hardware and software components that make up the VIEW system in order to facilitate system 
reconfiguration for changing research requirements. 

Before the beginning of the implementation phase of the Virtual Environment Workstation, a 
number of performance requirements and specifications for the major VIEW systems components 
were determined. These are discussed with respect to attaining an overall performance objective of 
providing 640x480 pixel resolution imagery at 30 frames per second over dual, independent, syn- 
chronized display channels with image viewpoints updated in coordination with head and limb 
motion. 

3. VIEW Implementation: Unique Issues and Problems 

The VIEW project attempts to match media and computational technology as closely as possible 
to the perceptual and cognitive capabilities of the human operator in order to achieve a state of 
Telepresence in which a sufficient quantity and quality of sensory feedback is presented to the opera- 
tor to approximate presence at a remote task site or in a synthetic environment. The factors that 
directly influence and effect the achievement of this display configuration are divided into three 
main areas and are discussed in detail as examples of problems and challenges for the development 
of virtual environment systems. 
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3.1 Computational issues. 

- Problems in achieving VIEW objective of 30Hz frame rate: 

- Overall cumulative system transport delay 

- “Realtime” UNIX 

- Communication timing problems and scheduling artifacts 

- Problems with original performance benchmarking 

- I/O problems and requirements: 

- RS232 requirements and host modifications 

- Parallel communication between host and graphics system 

- Floating point accuracy requirements in database representation. 

- System calibration requirements for research applications (tracker, etc.) 

- The need for evolution to object-oriented software structure for VIEW. 

3.2 Graphics implementation issues: 

- Requirements and problems in synching dual channel graphics systems for proper 
stereocopic image representation: 

- SGI IRIS graphics 

- HP SRX/TurboSRX graphics 

- ISG graphics 

Differences between human eye and virtual camera image geometry: 

- Viewport specification 

- Perspective transformations (world rotation) 

- Field of View specification 

- Display calibration requirements 

- Matrix/Euler/Quatemian representation; difficulties and advantages. 

- Graphics language implementations and extensions (GOSAMR) 

- Object editor 

- Problems with animating textures 

- Needed extensions 

- Idiosyncrasies of lighting model in VIEW environment. 

- Software structure for 3D information windows in VIEW environment: differences 
between screen space and world space. 

3.3 Human Factors issues (Telepresence requirements): 

- Guidelines and requirements necessary to provide stereoscopic images that are 
geometrically correct representations of the depth relationships and perspective of a 
rendered or captured scene are discussed (orthostereoscopy). For example, in an 
orthostereoscopic scene, an object in the Virtual Environment wpuld subtend the same 
angle of visual field as the object it represents in the real world and would also impart 
correct motion parallax and motion perspective cues in relation to its virtual 
surroundings as the viewer changes position. 
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- Differences between human eye and virtual camera image geometry are discussed with 
reference to view representation, depth of field representation, image separation for 
stereoscopic fusion, window effect in stereoscopic image fusion, foveal image display, 
and correct focal length and viewport specification. 

- Differences between human eye and virtual camera image QUALITY such as 
luminance, resolution, color, and texture are discussed as well as differences between 
human visual system and virtual camera image DYNAMICS such as frame rate and 
image generation lag, axis of rotation for changing viewpoint (tracker offset), image 
and spatial distortions from non-parallel optical axes, and infinity optics versus variable 
accomodation performance. 

- Multisensory virtual representations provide an additional dimension to the 
telepresence experience and are discussed in relation to synchronization with visual 
imagery, auditory localization and 3D sound cues, tactile feedback, and visceral 
simulation with motion platforms. 
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VIRTUAL ACOUSTIC DISPLAYS 
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As with most research in information displays, virtual displays have generally emphasized visual 
information. Many investigators, however, have pointed out the importance of the auditory system as 
an alternative or supplementary information channel (e.g., Deatherage, 1972; Doll, et. al., 1986; 
Patterson, 1982; Caver, 1986). A three-dimensional auditory display can potentially enhance infor- 
mation transfer by combining directional and iconic information in a quite naturalistic representation 
of dynamic objects in the interface. Borrowing a term from Gaver (1986), an obvious aspect of 
“everyday listening” is the fact that we live and listen in a three-dimensional world. Indeed, a pri- 
mary advantage of the auditory system is that it allows us to monitor and identify sources of infor- 
mation from all possible locations, not just the direction of gaze. This feature would be especially 
useful in an application that is inherently spatial, such as an air traffic control display for the tower or 
cockpit. A further advantage of the binaural system, often referred to as the “cocktail party effect” 
(Cherry, 1953), is that it improves the intelligibility of sources in noise and assists in the segregation 
of multiple sound sources. This effect could be critical in applications involving encoded nonspeech 
messages as in scientific “visualization,” the acoustic representation of multi-dimensional data (e.g., 
Bly, 1982) and the development of alternative interfaces for the visually-impaired (Edwards, 1989; 
Loomis, et. al., 1990). Another aspect of auditory spatial cues is that, in conjunction with other 
modalities, it can act as a potentiator of information in the display. For example, visual and auditory 
cues together can reinforce the information content of the display and provide a greater sense of 
presence or realism in a manner not readily achievable by either modality alone (Colquhoun, 1975; 
Warren, et. al., 1981; O’Leary and Rhodes, 1984). This phenomenon will be particularly useful in 
telepresence applications, such as advanced teleconferencing environments, shared electronic 
workspaces, and monitoring telerobotic activities in remote or hazardous situations. Thus, the com- 
bination of direct spatial cues with good principles of iconic design could provide an extremely pow- 
erful and information-rich display which is also quite easy to use. 

This type of display could be realized with an array of real sound sources or loudspeakers (Doll, 
et. al., 1986; Calhoun, et. al, 1987). An alternative approach, recently developed at NAS A-Ames, 
generates externalized, three-dimensional sound cues over headphones in realtime using digital sig- 
nal-processing (Wenzel, et. al., 1988a). Here, the synthesis technique involves the digital generation 
of stimuli using Head-Related Transfer Functions (HRTFs) measured in the two ear-canals of indi- 
vidual subjects (see Wightman and Kistler, 1989a:. Up to four moving or static sources can be simu- 
lated in a head-stable environment by digital filtering of arbitrary signals with the appropriate 
HRTFs. This type of presentation system is desirable because it allows complete control over the 
acoustic waveforms delivered to the two ears and the ability to interact dynamically with the virtual 
display. Other similar approaches include an analog system developed by Loomis, et. al. (1990) and 
digital systems which make use of transforms derived from normative mannikins and simulations of 
room acoustics (Genuit, 1986; Posselt, et. al., 1986; Persterer, 1989; Lehnert and Blauert, 1989). 
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Such an interface also requires the careful psychophysical evaluation of listeners’ ability to accu- 
rately localize the virtual or synthetic sound sources. For example, a recent study by Wightman and 
Kistler (1988b) confirmed the perceptual adequacy of the basic technique for static sources; source 
azimuth was synthesized nearly perfectly for all listeners while source elevation was somewhat less 
well-defined in the headphone conditions. 

From an applied standpoint, measurement of each potential listener’s HRTFs may not be possi- 
ble in practice. It may also be the case that the user of such a display will not have the opportunity 
for extensive training. Thus, a critical research issue for virtual acoustic displays is the degree to 
which the general population of listeners can obtain adequate localization cues from stimuli based on 
non-individualized transforms. Preliminary data (Wenzel, et. al., 1988b) suggest that using non-lis- 
tener-specific transforms to achieve synthesis of localized cues is at least feasible. 

For experiehced listeners, localization performance was only slightly degraded compared to a 
subject’s inherent ability, even for the less robust elevation cues, as long as the transforms were 
derived from what one might call a “good’’ localizer. Further, the fact that individual differences in 
performance, particularly for elevation, could be traced to acoustical idiosyncracies in the stimulus 
suggests that it may eventually be possible to create a set or “universal transforms” by appropriate 
averaging (Genuit, 1986) and data reduction techniques (e.g., principal components analysis), or 
perhaps even enhancing the spectra of empirically-derived transfer functions (Durlach and Pang, 
1986). 

Alternatively, even inexperienced listeners may be able to adapt to a particular set of HRTFs as 
long as they provide adequate cues for localization. A reasonable approach is to use the HRTFs from 
a subject whose measurements have been “behaviorally-calibrated” and are thus correlated with 
known perceptual ability in both free-field and headphone conditions. In a recently completed study, 
sixteen inexperienced listeners judged the apparent spatial location of sources presented over loud- 
speakers in the free-field or over headphones. The headphone stimuli were generated digitally using 
HRTFs measured in the ear canals of a representative subject (a “good localizer”) from Wightman 
and Kistler (1988a,b). For twelve of the subjects, localization performance was quite good, with 
judgements for the non-individualized stimuli nearly identical to those in the free-field. 

In general, these data suggest that most listeners can obtain useful directional information from 
an auditory display without requiring the use of individually-tailored HRTFs. However, a caveat is 
important here. The results described above are based on analyses in which errors due to front/back 
confusions were resolved. For free-field versus simulated free-field stimuli, experienced listeners 
exhibit front/back confusion rates of about 5 vs. 10 % and inexperienced listeners show average rates 
of about 22 vs. 34 %. Although the reason for such confusions is not completely understood, they are 
probably due in large part to the static nature of the stimulus and the ambiguity resulting from the so- 
called cone of confusion (see Blauert, 1983). Several stimulus characteristics may help to minimize 
these errors. For example, the addition of dynamic cues correlated with head-motion and well- 
controlled environmental cues derived from models of room acoustics may improve the ability to 
resolve these ambiguities. 
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Natural environments have a content, i.e., the objects in them; a geometry, i.e., a pattern of rules 
for positioning and displacing the objects; and a dynamics, i.e., a system of rules describing the 
effects of forces acting on the objects. Human interaction with most common natural environments 
has been optimized by centuries of evolution. Virtual environments created through the human-com- 
puter interface similarly have a content, geometry, and dynamics, but the arbitrary character of the 
computer simulation creating them does not insure that human interaction with these virtual envi- 
ronments will be natural. The interaction, indeed, could be supernatural but it also could be 
impossible. 

An important determinant of the comprehensibility of a virtual environment is the correspon- 
dence between the environmental frames of reference and those associated with the control of envi- 
ronmental objects. The effects of rotation and displacement of control frames of reference with 
respect to corresponding environmental references differ depending upon whether perceptual judge- 
ment or manual tracking performance is measured. 

The perceptual effects of frame of reference displacement may be analyzed in terms of distor- 
tions in the process or virtualizing the synthetic environmental space. The effects of frame of refer- 
ence displacement and rotation have been studied by asking subjects to estimate exocentric direction 
in a virtual space. Exocentric judgements involve the estimation of the orientation of one external 
object with respect to an external frame of reference. They may be contrasted with egocentric 
judgements in which objects orientation is judged with respect to the viewer himself. Though 
alternative models of the errors of exocentric direction may be compared, the most parsimonious 
explanation may be that viewers misjudge the direction of the view vector used to produce the 
display. The motor effects of frame of reference misalignment have been studied using a technique 
of three dimensional tracking in which a target moves irregularly in 3 axes. Pursuit tracking errors 
may be separated into linearly additive visual and visual-motor components, but the pattern of error 
as a function of angular misalignment may not generalize to compensatory tracking or to non- 
holonomic control modes. 
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NASA’s experience in planetary exploration has demonstrated that the desktop workstation is 
inadequate for many visualization situations. The primary mission displays for the unmanned Sur- 
veyor missions to the moon during the mid-1960s, for example, were environmental images assem- 
bled on the inside surfaces of spherical shells. Individual images taken in 1971 by the first Mars 
orbiter, Mariner 9, were applied to the outside of four-foot diameter spheres, creating composite pho- 
tographs of the entire planet. During the Viking mission to Mars in 1976, initial orbiter photos were 
mosaicked on large tables or on walls to serve as group decision-making displays. The environments 
surrounding the two Viking landers were visualized using stereo and panoramic images, desktop 
models, and full-sized replicas of the lander and the surrounding rocks. 

Future exploration missions will greatly benefit from advances in digital computer and display 
technology, but there remain unmet user interface needs. Unfortunately, the desktop workstation is 
generally considered to be the starting point of user-computer interface design, and most current 
human-computer interface research emphasizes the desktop metaphor. In this view, computing is 
thought to be comparable to paperwork, and the objects and functions available to the user reflect 
this emphasis. Point-and-click interaction with icons and windows is supportive of operations per- 
formed on text and simple pictures, but this approach is inadequate for planetary exploration. 

Alternative user interfaces and metaphors are needed for planetary exploration and other interac- 
tions with complex spatial environments. These interfaces and metaphors would enable the user to 
directly explore environments and naturally manipulate objects in those environments. Personal 
simulators, virtual workstations, and telepresence user interfaces are systems capable of providing 
this integration of user space and task space. The Exploration Metaphor is a useful concept for 
guiding the design of user interfaces for virtual environments and telepresence. To apply the Explo- 
ration Metaphor is to assert that computing is like exploration, and to support objects, operations, 
and contexts comparable to those encountered in the exploration of natural environments. The 
Exploration Metaphor, under development for user interfaces in support of NASA’s planetary explo- 
ration missions and goals, will also benefit other applications where complex spatial information 
must be visualized. 

Visualization methods and systems for planetary exploration are becoming increasingly inte- 
grated and interactive as computing technology improves. These advances will benefit from virtual 
environment and telepresence interface technology. A key development has been the processing of 
multiple images and other sensor data to create detailed digital models of the planets and moons. 
Data from images of the Earth, Mars, and Miranda (a moon of Uranus), for example, have been con- 
verted into three dimensional models, and dynamic virtual fly-overs have been computed as demon- 
strations. Similar processing of lower altitude photography and the use of computer aided design 
tools promise to produce very detailed models in the future. Currently, the Magellan spacecraft is 
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more than half way to Venus on a mission to digitize the entire surface of that planet, and this, too, 
will provide an interactive digital model for virtual exploration on Earth. 

Future plans call for multiple unmanned rovers to traverse the surface of Mars, enabling geosci- 
entists located at a central Mars base to explore the planet via telepresence. In addition, Earth-bound 
mission operators and scientists will be able to access integrated digital models of Mars as the mis- 
sions progress. The Exploration Metaphor will contribute to the design of user interfaces for these 
applications. 
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Early studies in our laboratory showed that cats could be trained to alter spontaneous EEG pat- 
terns through contingent food reinforcement of operant configurations. In particular, rhythmic EEG 
patterns in sensorimotor cortex, which were related behaviorally to motionlessness, were facilitated 
in this manner, and showed sustained alteration across states and time. Electrophysiological changes 
in the sensorimotor pathways were found to accompany this effect. Additionally, several striking 
behavioral changes were seen as well, including in particular an enhancement of sleep and an eleva- 
tion of seizure threshold to epileptogenic agents. 

This latter observation raised the possibility that human seizure disorders might be influenced 
therapeutically by similar training. However, in man the use of food reinforcement was not practical. 
Further, unlike our cats, human epileptics present with characteristic EEG abnormalities. An evalua- 
tion of these abnormalities from the perspective of our cat data showed that while normal humans 
had analogous patterns, untreated epileptics were often deficient in normal rhythmic sensorimotor 
activity and had EEG patterns dominated, instead, by slower and more poorly organized patterns. It 
thus became necessary to develop a very different strategy in applying this methodology to 
epileptics. 

Our objective in human EEG feedback training became not only the facilitation of normal 
rhythmic patterns, but also the suppression of abnormal activity, thus requiring complex contingen- 
cies directed to the normalization of the sensorimotor EEG. To achieve this, a multicomponent fre- 
quency analysis was developed to extract and separate normal and abnormal elements of the EEG 
signal. Each of these elements was transduced to a specific component of a visual display system, 
and these were combined through logic circuits to present the subject with a symbolic display. This 
display allowed the subject to score “points” in a variety of game formats through the normalization 
of EEG characteristics. Variable criteria provided for the gradual shaping of EEG elements towards 
the desired normal pattern. Some 50-70% of patients with poorly controlled seizure disorders experi- 
enced therapeutic benefits from this approach in our laboratory, and subsequently in many other lab- 
oratories as well. 

A more recent application of this approach to the modification of human brain function in our 
laboratory has been directed to the dichotomous problems of task overload and underload in the 
contemporary aviation environment. At least 70% of all aviation accidents have been attributed to 
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the impact of these kinds of problems on crew performance. The use of the EEG in this context has 
required many technical innovations and the application of the latest advances in EEG signal analy- 
sis. Our first goal has been the identification of relevant EEG characteristics. Additionally, we have 
developed a portable recording and analysis system for application in this context. Findings from 
laboratory and in-flight studies suggest that we will, indeed, be able to detect appropriate changes in 
brain function, and feed this information to on-board computers for modification of mission 
requirements and/or crew status. 

Supported by the Veterans Administration, the Public Health Service, NASA, and Northrop 
Aviation. 
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THE “TELEPRASENZ” CONSORTIUM: STRUCTURE & INTENTIONS 


JensBlauert 

Ruhr-Universitat Bochum 
Bochum, FR Germany 

The Teleprasenz-Consortium is an open group of currently 37 scientists of different disciplines 
who devote a major part of their research activities to the foundations of telepresence technology. 
The necessary support for their work is provided by German federal and state sources as well as by 
European agencies and industry. The activities of die consortium are organized into three main 
branches: virtual environment, surveillance and control systems and speech and language technol- 
ogy. In the following, a brief summary of the main activities in these areas is given. 

Telepresence technology - in this context - is basically understood as a means to bridge spatial 
and temporal gaps as well as certain kinds of concealment, inaccessibility and danger of exposure 
(e.g., microscopic dimensions, non-perceptibility by human senses, sensoric disabilities, speech and 
language problems, load of emissions). 

Virtual Environment: In this group, it is the aim to provide a virtual environment for specific 
tasks such as telesurveillance or telecooperation applications. Although the main focus was directed 
on the auditory and visual components so far, future work will include tactile and kinesthetic cues. 

Surveillance and Control Systems: This group is mainly engaged in tasks which concern 
control technology, diagnostics of complex systems in biology and technology as well as in 
accompanying problems in signal processing and software technology. 

Speech and Language Technology: This group is concerned with several projects in the areas 
of automatic speech in- and output, speech coding and symbolic processing fore telepresence 
scenarios. 

Current research proposals of the Teleprasenz Consortium include a telemanipulator system and 
a testbed for polysensory virtual environment as well as many smaller research and development 
contracts. An information brochure on Teleprasenz (in German) will be available at the conference. 
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MACHINE PERCEPTION AND ITS ROLE IN VIRTUAL ENVIRONMENTS 


S, K. Ganapathy 
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(No written material provided) 
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PANEL DISCUSSION ON TELEPRESENCE AND SENSORIMOTOR 

ADAPTATION 

R. B. Welch, Chair 
NASA Ames Research Center 

J. Loomis 

University of California, Santa Barbara 
R. Pepper 

Science Applications International, Inc. 

M. Kreuger 

Artificial Reality Corporation 
and 

T. Etter 
ANPAWest 



Four questions were addressed. The following represents a synopsis of the thoughts of each 
seminar member (or members) with respect to his/her assigned item. 

1. What do we mean by the term “telepresence” and how can we measure it? (Jack Loomis) 

If virtual and remotely sensed environments are comprised of the same sensory cues as near- 
sensed worlds (i.e., “equivalent configurations”), telepresence will be complete. This rather self-evi- 
dent observation leads to the much more interesting issue of why some everyday perceptual events 
are externalized (experienced as non-self), while others are not. Pain, for example, is perceived as 
internal, while most objects in our perceptual experience are externalized. 

It has been proposed that the degree of experienced extemalization is related to the range of 
variation in the efference to the motor system that results in some invariance in the sensory inflow 
(e.g., Brunswik, Gibson, Polanyi). According to this view, pain is completely internalized because 
the sensory signals are constant with changes in motor outflow. A positive afterimage produces 
intermediate extemalization because the change in its angular position within the visual field is 
strictly correlated with eye movements. The most compelling sense of extemalization arises when 
sensory input from all of the senses provides converging evidence, under active exploration, of some 
fixed entity. If this compelling extemalization arises when using a virtual display or teleoperator, we 
say that there is “telepresence,” but just as interesting is the extemalization that occurs with ordinary 
near-sensing of the world. 

Finally, a necessary condition for extemalization (telepresence) when using a virtual display or 
teleoperator is that the observer be capable of internalizing the dynamics of the linkage (plant) 
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mediating contact with the virtual or remote environment. If the linkage is too unnatural, arbitrary, or 
non-linear for this to occur, the observer will experience little extemalization of the distal environ- 
ment. Nevertheless, adaptation may serve to re-establish some sense of extemalization. Once this 
happens, the plant model becomes transparent to the observer and extemalization of the distal envi- 
ronment results. A common example involves long-cane training for the blind. At first, the trainee 
fumbles a lot and is very aware of the cane rather than the environment. As skill develops, however, 
manipulation of the cane becomes more automatic, resulting eventually in its transparency and in the 
extemalization of the environment with which the tip of the cane is making contact. 

2. Of what good is telepresence? Does it contribute to performance? Is it self-evident that the 
more telepresence, the better? (Ross Pepper) 

A number of benefits result when the experience of telepresence is increased. First, under height- 
ened telepresence, less operator training is necessary. Second, teleoperator performance under these 
conditions is more comparable to direct human performance. Third, optimal telepresence can be 
expected to increase the “emotionality” of the remote environment, depending on the content of the 
replicated sensory experience. An example of the latter point is the enjoyable visceral feelings of 
actually “being there” experienced by the viewers of a very realistic, action-filled movie. Finally, the 
very search for telepresence will stimulate the conceptual analysis of perceptual issues (see Loomis’ 
remarks). 

Telepresence contributes to performance, but not without certain costs. The benefits of telepres- 
ence to performance are that it (1) reduces errors and task performance variability, (2) increases pre- 
cision,. (3) reduces training costs, (4) enables performance similar to direct viewing conditions, and 
(5) results in greater adaptability to changing task conditions. 

The costs of enhanced telepresence are (1) an increase in system complexity, (2) reduced relia- 
bility, (3) increased initial investment, and (4) increased maintenance costs. 

It may be concluded that teleoperator system enhancements that produce marked increases in 
performance generally result in a heightened experience of telepresence. Examples of such 
enhancements are changing a monocular visual display to a stereoscopic display and providing force 
feedback for the manipulating hand. 

3. Which characteristics of the teleoperator or virtual environment system contribute to 
telepresence, which are irrelevant, and which degrade it? (Myron Krueger and Tom Etter) 

(Krueger’s remarks) 

A number of factors influencing telepresence or artificial worlds can be addressed, the most 
influential of which is maintaining a short delay between action and response. This is more important 
than a high sample rate by means of pipelining techniques. It is also claimed that short delays are 
more important than a realistic portrayal in graphic worlds. The allowable delay varies with the task. 
In particular, fine manipulation requires shorter delays than does gross manipulation. The ability to 
look around suffers with delays longer than .03 seconds; it can be done, but the operator is aware of 
the lag. In general, there is a decline in possible telepresence with distance. Beyond a rather short 
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distance, the only way the immediacy of telepresence can be attained is by interacting locally with a 
graphic world. 

Another dimension that may be explored is the effect of unencumbering sensors and actuators 
designed to increase telepresence. The first observation is that the presence of encumberance signals 
a distance from the world in which one wishes to feel present. Also, the discomfort caused by pro- 
longed encumberance can lead to a net loss in telepresence. 

It should also be observed that realism may not necessarily be the ultimate goal of telepresence. 
Natural behavior can be instantly achieved in 2-D realities which are easier to enter and exit than are 
3-D worlds than can only be experienced by “suiting up.” Furthermore, if artificial worlds are to be 
used for applied purposes, many traditional techniques will be demanded that violate realism. While 
people may initially obey the spatial metaphor to go from one point in a virtual world to another, 
later they will probably want to make the trip instantly by means of virtual teleportation. 

A final event that will affect telepresence in artificial realities, as well as in tasks where two 
telerobots cooperate at a remote site, is the representation of the operator at the effector end. If the 
individual is to be represented, it is inevitable that a face must be provided and no method of three- 
dimensional facial image capture and portrayal currently exists. 

(Etter’s remarks) 

What enhances telepresence? Here’s a related question: What enhances the “realism” of artificial 
food? In this case we’re clearly dealing with two separate issues, taste and food value. I suggest that 
essentially the same two issues arise in the case of “artificial presence.” We must distinguish the 
sense of presence from the physical component of presence, whatever exactly that may be. Suppose 
that you are wired into a computer-generated virtual environment. Does the computer supply this 
physical component of presence? Do the wires carry it? 

Bell’s theorem shows that any informational model of quantum phenomena (e.g., computer 
simulation) must contain faster-than light information paths, thereby contradicting the theory of 
relativity. Thus, if relativity does in fact govern the transmission of information, the connections 
among things in the world cannot be purely informational — they must also transmit something else. 
Let’s call this something else the L factor. 

The L factor involves negative probabilities which makes its meaning quite mysterious. How- 
ever, its mathematics are very simple and it leads to an elegant and robust explanation of basic 
quantum theory. The only alternative to the L factor is to assume the existence of hidden, non-local 
information paths. This is easy to interpret in familiar ways, but it fills the world with unobservable 
entities and generally creates a mess. 

Assuming that we go with the L factor, what might this mean for telepresence? Could the L fac- 
tor be the “food value” of physical presence? Could there be a long-term danger in going without it, 
like eating food with vitamins? With a more scientifically advanced technology, might we be able to 
build virtual environments that supply it? 
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4. To what extent is it necessary for the human operator to adapt to the teleoperation or vir- 
tual environment situation in order to experience a strong sense of telepresence and/or to 
function effectively? (Robert Welch) 

With current technology, remotely sensed and virtual worlds inevitably entail significant inter- 
sensory and sensorimotor discordances, which serve to degrade both performance and the experience 
of telepresence. However, it is likely that these problems can be overcome, at least partially, by 
means of adaptation or adaptation-like processes, of which there are four varieties: (1) intersensory 
bias, (2) perceptual adaptation, (3) perceptual-motor adaptation, and (4) habituation. 

The most commonly cited example of intersensory bias is ‘Visual capture”— the “dominance” of 
vision over felt limb position. It represents an immediate and very short-lived resolution of the inter- 
sensory conflict since it disappears as soon as vision is precluded. The magnitude of intersensory 
bias is inversely related to the size of the intersensory discrepancy and it is facilitated by familiarity 
with the intersensory relationship (e.g., a view of the hand) as well as by the presence of congruent 
intersensory characteristics of the stimulus object such as its felt and seen shape, motion, orientation, 
and size. These same factors are presumably involved in creating a compelling experience of 
telepresence. 

Perceptual adaptation represents a compensatory recalibration of one or both of the sensory sys- 
tems that serves to resolve the intersensory discrepancy. In contrast to intersensory bias, it persists as 
an aftereffect. This type of adaptation will not occur with very large discrepancies, such as 
180-degree optical rotation. 

Perceptual-motor adaptation refers to changes in the observer’s performance. If perceptual adap- 
tation has occurred, perceptual-motor adaptation is likely to follow suit, whereas the reverse does not 
always hold, particularly with very large rearrangements. For example, even though after several 
days of practice one will get quite good at operating in a 180-degree rotated visual world, it will 
continue to look upside down. 

Factors that affect the magnitude of perceptual and/or perceptual-motor adaptation are active 
(versus passive) movement, immediate feedback, error correction, distributed practice, and incre- 
mental exposure. Presumably, these variables would similarly influence adaptation to rearranged 
remote or virtual environments. 

After prolonged experience with the rearranged sensory environment, it will come to look famil- 
iar, a phenomenon referred to here as habituation. This may occur even when no perceptual adapta- 
tion has taken place. For example, well practices wearers of inverting goggles become so accus- 
tomed to their upside-down visual world that for periods of time they pay no attention to its orienta- 
tion. It is likely that at those times the feeling of presence is strong and that after much practice the 
same would be true for a remote or virtual world that may have initially appeared quite discrepant or 
bizarre. 

Finally, if and when it is possible to construct a teleoperator or virtual environment system that is 
completely accurate in its portrayal of the naturally sensed world, there will be little to which the 
operator will have to adapt. However, even with such an ideal system, some adaptation will be 
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necessary to overcome the inter-operator differences in perception and reponse that have been 
widely reported. 
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PANEL DISCUSSION ON PERFORMANCE MEASUREMENT AND 

EVALUATION METHODS 


Daniel T. Ling, Chair (IBM), Ruzena Bajcsy (University of Pennsylvania), Anatal Bejczy (Johnson 
Propulsion Laboratory), Beth Marcus (Exos, Inc.), and Misha Pavel (Stanford University) 


The seminar was introduced by Daniel Ling who presented a brief outline of a number of key 
issues which need to be addressed within the context of performance measurement and evaluation 
methods. 

1. The context of any evaluation must be the task. How should the tasks be defined and 
decomposed? 

2. What needs to be measured? This really means obtaining the data needed to develop models. 
Models are needed for both the human operator as well as the hardware/software system. 

3. What criteria should be used for the measurements? This might include task performance (mean 
and variance), training required, ease of recall, “likeability” which may NOT correspond to any 
simple measure of performance, etc. 

4. Need for rigorous experiments as well as general observations and rules of thumb. 

Misha Pavel advocated evaluations based upon explicitly specified system models. He also dis- 
cussed the importance of task analysis. 

Model Based Evaluation (Misha Pavel) 

Evaluation of interactive human-machine systems is usually very difficult because there are too 
many variables, too many conditions and the results obtained with one system are often difficult to 
apply to other similar ones (generalization problem). Frequently, the only way to make this problem 
manageable is to formulate an explicit “computational” model of all the components of the system 
including the human operator. Such a model, because of the constraints it imposes, will permit con- 
siderably more efficient evaluation. For example, many formal models specify a functional relation- 
ship among variables providing only a few free parameters. Then, instead of evaluating every possi- 
ble combination of all the variables, one must only determine a smaller set of parameters. 

Whenever performance of a system is measured, there is an underlying model even if it is not 
explicitly specified. These implicit models are usually weaker in that they impose fewer constraints. 
They are therefore more general and have less predictive power. The distinction between weak and 
strong models is useful. A strong model imposes strong constraints on the system behavior, and 
therefore allows for very efficienct evaluation. Its disadvantage is that it is likely to be incorrect. A 
strong model, for example, would have a second order polynomial relationship between two 
variables. A weak model would more likely be correct but would not allow as efficient evaluation. 
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For example, a very weak model would only require a monotonic relationship between two 
variables. This model is less constraining, but makes less accurate predictions. 

Task Analysis 

Specification of a model has further advantage in that it requires a careful task analysis. The 
analysis alone can frequently reveal design problems before a prototype is actually built. For exam- 
ple, task analysis can reveal whether the systems objectives are well defined. Objectives are neces- 
sary in order to test and evaluate any system. Too often we see evaluation of a system with no par- 
ticular objective specified to the operatory. The generalizability of such results is questionable 
because in real-world situations the operator always has a goal to accomplish. 

In some cases, the system is designed for many purposes (e.g., a teleoperator for extraterrestrial 
applications) and it is difficult to specify a single task. In those situations one should estimate the 
envelope of the performance capabilities and analyze several extreme tasks. For example, in the case 
of a teleoperator design, one might expect tasks of grasping and moving large objects, executing a 
complex trajectory, and, at the other extreme, inserting a small fastener into a hole. 

One of the results of careful task analysis is the determination of the variables to be measured. 
Such analysis might reveal the need to assess other measures than the usual duration of a given task. 
In particular, an important by-product of a task analysis is the notion that the operator usually has a 
choice of strategies. A typical, simple example is the speed-accuracy trade-off arising in evaluation 
of various input devices. In interactions between a human and a machine most tasks can be per- 
formed faster by sacrificing accuracy. A simple, weak model, of the speed-accuracy trade-off leads 
to the following evaluation principles: 

1 . Always measure both speed and accuracy. 

2. Assure sufficient time pressure on the operator so that their performance has measurable 
errors. 

The second principle assures that the experimenter can detect if the operator changes strategy 
between two conditions or two systems. 

While I have indicated the need for model-based evaluation, it is important to note that using 
models can be very difficult. One argument frequently used against the model-based approach is that 
the validation of the model is by itself a complex empirical problem requiring too large an experi- 
mental effort to be justified. Although this is a serious problem the cost of evaluation must be com- 
pared to the savings from the ability to generalize the conclusions and to debug the system. 

Ruzena Bajcsy echoes the theme of model based evaluation and task analysis. 
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Notes (Ruzena Bajcsy) 

I am most interested in sensory system performance. The way I view performance evaluation is 
that it implies: 

Models 

By models, I mean characterization of the physics and geometry of the sensory system. Engi- 
neers have been doing modeling for a long time. For example, for a visual sensor we must model its 
optics, absorption functionl, resolution, signal to noise ratio, etc. Similarly for pressure/force sensors 
we should model sensitivity, saturation, hysteresis, and the like. For manipulators, the models give 
degrees of freedom, the reach-space, calibration parameters, repeatability, accuracy, etc. In other 
words, the models provice for each parameter the limits of performance. What is not so obvious is 
that similar modeling principles must apply to software modules as well as to hardware modules. 
Examples of these software modules are: convolution (parameter is width and orientation of the fil- 
ter), edge-detector (looking for the first derivative or second derivative, the parameter is threshold of 
acceptance of the peak in the first derivative or second derivative), various fitting procedures to fit 
surfaces, volumes and so forth. This is also related to the choice of primitives which serve as data 
reduction mechanisms, and which are necessary for displays, and/or other communication means. 
These primitives are in fact more sophisticated “sensors” implemented in software. 

Models of the Task 

While models of the sensors tell us what CAN be measured and detected, a model of the task will 
tell lus what NEEDS to be measured. The bottom line is, that by the choice of models, one artifi- 
cially constrains the problem, but this is the only way to have a predictable system where you take as 
much data as you NEED, determined by the TASK, and you explain the data/measurements as much 
as the models allow. So evaluation of a system is always in terms of some a priori chosen models. 

Antal Bejczy described specific issues which need to be evaluated for teleoperator systems in 
space applications. He also emphasized task analysis and decomposition and an understanding of the 
target user. 

Measurement and Evaluation in Teleoperator Systems (Antal Bejczy) 

Virtual environments for teleoperator control will play an increasingly important role as the sys- 
tem and task complexity and telecommunication time delay increase. Preview and predictive graph- 
ics displays for task planning and time-delayed control execution can create an essentially virtual 
environment for the operator. When the graphics images of the manipulator and eventually also of 
elements of the task environment are overlayed over the real TV images of the manipulator and task 
environment, the performance capabilities of this human-machine interface technique will essentially 
depend on the fidelity of calibration by which the graphics images are related to the real images. This 
is an objective measure, and depends on the knowledge of a set of system parameters (TV camera 
parameters), on mathematical calibration techniques, and on the resolution of graphics images. The 
higher the calibration fidelity, the higher is the operator’s confidence in the preview and predictive 
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power of the graphics. Through high-fidelity calibration techniques, the graphics images of manipu- 
lator and task environment can truly become “virtual realities” to the operator. 

Graphics displays of abstractions in teleoperator control are essentially visualizations of invisible 
events or situations. Such events or situations are: forces exerted by manipulator on objects and on 
environment; workspace constraints or “holes” in a workspace not accessible under certain task 
conditions; etc. They provide guidance to the operator. Their performance essentially depends on 
their formats and on the easy mapping of the display format to the task environment. Here, pointed 
simplicity plays a key role. The development of such displays is in an embryonic state. 

In many cases, human-machine interface devices and techniques in teleoperation are evaluated as 
separate entities or “subsystems,” and their full-valued performance is predicted from such isolated 
evaluations. Such evaluation methods are useful when different interface devices and techniques are 
compared. However, a specific device or technique best reveals its virtues or weaknesses when eval- 
uated in a full system operation setting. 

In many cases, the “naive user” is used as a “reference operator” forjudging or evaluating a 
human-machine interface device or technique in teleoperator control. Though tins idea has a well- 
motivated intention, care should be taken to develop a more tangible definition of the “naive user” as 
a “reference operator.” Probably what we mean by a “naive user” when we judge and evaluate a 
human-machine interface device or technique is: the “natural appeal” of the technique, the easy-to- 
remember features of the device, the amount of learning and training needed to be fully familiar with 
the device and technique, and so on. The more advanced human-machine interfaces do indeed 
require familiarization and training before their actual use. In particular, when teleoperation becomes 
more and more computer aided, the control station will acquire a more and more “abstract” ambience 
which will have to be mapped to the mechanical world. 

The case of multiple operators operating a teleoperator system received little attention from the 
viewpoint of performance measurements and evaluation methods, though it is a fact that many tele- 
operator systems are operated by at least two operators. This is particularly true for dual-arm teleop- 
erator systems. The obvious questions are: workload and function distribution, (pilot-copilot; driver- 
navigator; etc.), and the distribution of responsibilities. A good evaluation method for multiple oper- 
ator teleoperated systems could also pave the way for defining ways to convert such systems into 
single-operator systems. 

Beth Marcus discussed the notion that evaluations must be task and performance oriented rather 
than simply based on an abstract notion of “telepresence” or fidelity to the real world. 

Comments on Evaluation (Beth Marcus) 

The bottom line from the panel discussion is that evaluation of human interfaces is not easy. The 
suitability of a given interface for teleoperators or virtual environments is often very task specific. 
However, it is possible to begin to collect a “wish list” for these devices. All other things being equal 
the accuracy of sensing should be maximized, tr ainin g time minimized, and task performance maxi- 
mized. In addition, the trade-off between performance (i.e., speed, accuracy) and training time 
should be considered. 
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It is important to note that, since no interface hardware today or in the foreseeable future pro- 
vides all types of feedback and gives a feeling identical to “being there,” we must throw away the 
notion that “natural” equals good. In a non-ideal world, one might be able to convey more informa- 
tion in a more useful form using an artificial presentation. For example, human tactile sense gives no 
information about proximity, though it may be easy and useful to convey this information through a 
tactile display. 

In order to rapidly expand the utility of virtual environments, we need to take some of the tech- 
nology that exists in laboratories today and commercialize it to solve real-world problems. Thus, by 
providing low cost interfaces which advance beyond the capabilities of a mouse or a keyboard, we 
can have 10,000 or 10 million people contributing to the knowledge base rattier than a few hundred. 
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PANEL DISCUSSION ON DESIGN PRINCIPLES AND PREDICTIVE 

MODELS 


H. L. Alexander, Chair (MIT), N. Hogan (MIT), J. M. Hollerback (McGill University), 

I. Hunter (McGill University), and W. Verplank (I. D. TWO) 


Questions Discussed: 

1 . When should master compliance be stiff and when soft? When should slave compliance be 
stiff and when soft? What are the appropriate places for compliance? When should dampling 
be added to master or slave? 

2. When is it necessary to have an intelligent-computer-mediated telerobot, and when will direct 
teleoperation suffice? What functions are appropriate to the computer mediator? 

3. What machine sensory capabilities are important, in order to support increased computer 
participation in manipulation? Conversely, what remote human sensory modes are important, 
and how best to present sensory data to humans? 


4. Do current robot designs tend to be too anthropomorphic, and how should we work to design 
the task (and environment) for the sake of the robot as opposed to designing the robot? 

The seminar opened with comments by the panel participants. Bill Verplank introduced the use 
of metaphors to help drive the design process. As an example, he discussed the relatively familiar 
desktop metaphor for the computer interface, introduced at Xerox PARC, that has suggestd many 
aspects of modem interfaces, such as the organization of files (documents) into sets (folders) that can 
be handled, copied, moved, and displayed analogously to file folders in and on a physical desk. 

Bill Verplank introduced a further metaphor, motivated by the computer’s functions that may be 
shared with clothing, as opposed to the desk. He noted that the functions of clothing include physical 
protection, providing a sense of belonging or membership (as in the case of uniforms or school col- 
ors), and generating a sense of satisfaction in ones appearance. The point, again, is to motivate think- 
ing into how a computer might serve the same functions as clothing, and perhaps in ways similar to 
clothing. 

Neville Hogan addressed the adaptation of unfamiliar remote vehicle dynamics, such as the 
gravity-free dynamics of a teleoperated space vehicle or the viscosity-dominated dynamics of 
underwater vehicle, in order to present the operator with more natural dynamics to which he/she is 
accustomed. The point is to adapt the dynamics of the remote environment, through computation and 
control, to the operator, rather than compelling the operator to adapt his/her self to the remote- 
environment dynamics. 
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Neville also addressed the idea of turning manipulator dynamics to tasks, pointing out that 
humans actually do this relatively seldom, and generally by a modification of posture (for example, 
in increasing stiffness by strengthening the arm). Antal Bejczy spoke up to defend the use of selec- 
tive compliance in the performance of manipulation tasks. 

John Hollerbach spoke on the design of manipulators, both macro- and micro-, stressing the 
advantages of parallel-link manipulators in terms of their motion-error behavior and their stiffness. 
He pointed out that in a parallel-link manipulator, individual errors tend to average rather than to add 
as in the case of serial-link manipulators, leading to an overall decrease in positioning errors. 

Tom Sheridan responded to this, mentioning the differing characteristics of the Jacobian matrices 
corresponding to parallel- and serial-link manipulators. He went on to introduce a new topic, and a 
new term, having to do with the design of human-machine interface, for virtual and teleoperator 
interfaces. He coined the term “haptic teleproprioceptive psychophysics” to refer to the detailed 
study of human perception of manipulation cues that result from contact with objects. He pointed out 
that it is important to understand the details of human proprioceptive responses, so that we can take 
full advantage of that understanding in delivering those cues to the operator, as though force-reflect- 
ing hand controllers.* He pointed out that input from persons studying that field, and their participa- 
tion in meetings such as this one, might greatly benefit teleoperation and virtual environment studies. 

Ruzena Bajcsy emphasized the need for much further sensor development, to allow for the 
detection in the remote environment, of tactile information for reflection to the operator. This led to a 
discussion of sensor and actuator design, with particular reference to micromachining and its poten- 
tial in the development of tactile sensors and displays. The group discussed micromachining and its 
potential for creating actuators able to exert the forces and displacements necessary to excite human 
touch sensors. Handayam SriniVasan’s work in tactile perception sensitivity (which shows that 
human touch is indeed very sensitive) indicates that such actuators may not be too far out of reach. 

This discussion, focusing as it did on the precision of feedback to individual fingers from a dex- 
trous remote manipulator, led to discussion of alternatives, particularly of specialized remote tools to 
be used in place of general purpose remote manipulators. Nat Durlach prompted lively discussion by 
inviting designs of a manipulator that improve on the human hand. Two such were mentioned - 
canoe carvers with no pinkie fingers to get in the way, and pianists who have cross connections in 
their digital tendons severed in order to improve the independent mobility of their fingers. Unfortu- 
nately, neither of these manipulators, however improved over the originals, lend themselves to 
robotic applications at the current time. 

Such consideration, however, led to die discussion of the use of muscle fibers as robotic drive 
elements. This, in turn, introduced discussion of the phase-transition (memory) characteristics of 
Nitinol wire in the development of actuators displaying high stiffness and force capabilities. It was 
also suggested that skin grown in vitro might be used for tactile sensing with afferent nerves appro- 
priately connected to electronic sensors and the resulting nerve impulses processed by neural 
networks. 

With regard to specialized tools replacing general-purpose dextrous end-effectors, the point was 
made that individual tools may be developed that are much better adapted to specific manipulation 
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and other tasks, then are currently feasible dextrous manipulators. A general-purpose tool interface, 
rather than a dextrous end-effector, would serve to attach each such tool to the manipulator wrist and 
to conduct the appropriate interface services and signals. One such swappable end-effector would of 
course be a semi-dextrous hand, ranked next to the drill, grinder, wrench, saw, and nut driver. The 
chairman proposed developing a general-purpose operator “tele-handle” for a teleoperator master 
arm, that would provide a general-purpose interface to remote or virtual tool, and could supply a 
wealth of tactile and other feedback to the operator. 

Teleoperator tactile feedback having been well-discussed, the subject of tactility in virtual worlds 
was addressed. Whereas in teleoperator interfaces to remote, physical worlds the problem includes 
both sensing (remotely) and generating (locally) tactile stimuli, in virtual worlds the sensing problem 
is eliminated. The problem becomes instead one of synthesizing tactile stimuli, corresponding either 
literally (as in the case of contact with a hand, unyielding virtual wall) or metaphorically (what does 
it feel like to have ones hand pass through a virtual wall?) to the virtual reality. 

♦Chairman’s note: With our increased understanding of and capability for tactile feedback, we 
may want to generalize the term “force-reflecting” to sense-reflecting hand-controllers. An example 
is Steve Jacobson’s controller which delivers the tap responses to the operator on contact with 
objects, which response is not merely derived from force feedback, but is deliberately synthesized to 
represent to the operator the contact event. 
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MINI SEMINAR: TELEOPERATORS WITH MICRO SLAVE ROBOTS 




K. J. Gabriel, Chair 
University of Tokyo 


102 



POSTER PRESENTATIONS 




N95- 14040 

THE HUMAN OPERATOR TRANSFER FUNCTION: IDENTIFICATION OF 

THE LIMB MECHANICS SUBSYSTEM 

Lynette A. Jones and Ian W. Hunter 
McGill University 
Montreal, Quebec, CANADA 


In most studies of human manual control systems, the operator is characterized as a lumped, 
indivisible system, whose dynamics are linear and of low order. The objective of our research is to 
decompose the performance of the human operator in terms of the subsystems that determine the 
operator’s responses in order to establish how the dynamics of these component subsystems influ- 
ence the operator’s performance. In the present experiment, the dynamic stiffness of die human 
elbow joint was measured at rest and under different levels of biceps muscle activation; this work 
forms part of the analysis of the limb mechanics subsystem. 

The stiffness of the elbow joint was determined by subjecting the right wrist to stochastic posi- 
tion perturbations (S.D. 3mm, bandwidth of 100Hz) delivered by an electromagnetic linear motor 
(100 mm maximum linear displacement, peak static force 500 N). Subjects generated a target force 
(ranging from 0 to 150 N) during the perturbations, and were provided with visual feedback of the 
mean force produced. 

The stiffness frequency response function was determined at each mean force level. These func- 
tions were calculated by Fourier transforming the stiffness impulse response functions. Figure 1 
shows the stiffness frequency response functions obtained from a subject generating a mean force of 
25N (thin line) and 150N (thick line). The increase in the low frequency stiffness with an increase in 
mean muscle activation level (as reflected in the mean force) is clearly evident. The figure also 
demonstrates that stiffness increases at higher frequencies. This essentially high-pass characteristic 
of the limb mechanics system is well-known. In Figure 2, a typical coherence function is shown. The 
decrease in coherence at higher frequencies is due, in part, to a loss in the signal-to-noise ratio 
caused by limitations in the bandwidth of the motors, whereas the loss of coherence at lower fre- 
quencies probably reflects the contribution of reflexes. 
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We present here some of the salient results on the tactual discriminabilities of human subjects 
obtained through psychophysical experiments, and the associated peripheral neural codes obtained 
through electrophysiological recordings from monkey single nerve fibers. Humans can detect the 
presence of a 2 microns high single dot on a smooth glass plate stroked on the skin, based on the 
responses of Meissner type rapidly adapting fibers (RAs). They can also detect a 0.06 microns high 
grating on the plate, owing to the response of Pacinian corpuscle fibers. Among all the possible rep- 
resentations of the shapes of objects, the surface curvature distribution seems to be the most relevant 
for tactile sensing. Slowly adapting fibers respond to both the change and rate of change of curvature 
of the skin surface at the most sensitive spot in their receptive fields, whereas RAs respond only to 
the rate of change of curvature. Human discriminability of compliance of objects depends on 
whether the object has a deformable or rigid surface. When the surface is deformable, the spatial 
pressure distribution within the contact region is dependent on object compliance, and hence infor- 
mation from cutaneous mechanoreceptors is sufficient for discrimination of subtle differences in 
compliance. When the surface is rigid, kinesthetic information is necessary for discrimination, and 
the discriminability is much poorer than that for objects with deformable surfaces. 
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ABSTRACT 


This summary focuses on the kinesthetic interface of a virtual invironment system that was 
developed at the Newman Laboratory for Biomechanics and Human Rehabilitation at M.I.T. for the 
study of manual control in both motorically impaired and able-bodied individuals. 


MECHANISM DESIGN 


The kinesthetic interface is an electromechanically actuated manipulandum (i.e., backdriveable 
manipulator) that enables dynamic interaction between the arm of a human operator/experimental 
subject and controlled mechanical loads. The manipulandum, depicted in Figure 1, consists of a 
direct-drive handle linkage which is coupled to two printed armature brushed DC motors. The inter- 
face is instrumented to sense displacements, velocities, and accelerations at the motor shafts, and 
forces between the hand grip and the linkage. 

The coupling mechanism that joins the handle to the motors forms a spherical closed-chain five- 
bar linkage, resulting in two kinematic degrees of freedom at the hand grip. This linkage permits 
both actuator housings to be anchored to a common mechanicsl ground, thereby making it unneces- 
sary to carry their excessive weight and inertia. Because the linkage consists of only ball bearing 
revolute pairs and rigid links, undesirable friction, backlash, and compliance characteristics associ- 
ated with other transmission types are greatly reduced in this interface. 

Additional attributes of this mechanism configuration prove advantageous for manipulandum 
control. Over its 17 cm square workspace, the interface is approximately planar, and, most impor- 
tantly, the actions of the two motors are nearly decoupled. 


SYSTEM CONTROL 


The overall virtual environment system, shown in Figure 2, consists of the kinesthetic interface 
and a planar video display. System operation is supervised by an LSI-1 1/23 based micro computer, 
responsible for collecting and storing data, commanding and monitoring the Control Interface Unit 
(CIU), and transferring target and tracking response information to the Amiga 1000 personal 
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computer for display to the subject. The CIU contains hardwired circuitry to control load simulations 
by each motor as well as maintain system safety. 

Because of the manipulandum’s nearly planar and nearly decoupled linkage kinematics, configu- 
ration space control can be used to regulate endpoint impedance directly, without geometric compu- 
tation. As a consequence, the LSI- 1 1/23 host can update CIU load control parameters at rates 
exceeding 1 kHz. 


MECHANICAL LOAD AND OBJECT SIMULATION 


The manipulandum’s ability to produce positive {i.e., resistive) and negative (i.e., destabilizing) 
spring and damping fields was examined. Load ranges achieved with the CIU’s hardwired digitally- 
supervised analog controller are listed in Table 1. Greater stiffness — -up to 3 times higher — are 
obtainable under purely digital control implementations. 


Load Type 

Rotational 

Translational Equivalent 

Active Resistive Stiffness 
Active Resistive Damping 
Active Inertia 
Maximum Torque/Force 

0-136 N-m/rad 
0-6.7 N-m/rad/s 
.007-0.045 kg-mZ 
4.7 N-m 

0-3100 N/m 
0-152 N/m/s 
.13-1 kg 
21 N 


Table 1 : Load ranges achieved under digitally supervised analog control. The translational load 
equivalent is calculated for the hand grip end of die fixed length handle. 

In addition to simulating mechanical impedances, the controller can either mask or enhance the 
manipulandum’s passive inertia and friction, thereby allowing the handle to feel “invisible” when 
pushed and have sufficient friction to stay in place when “let go” by the human operator. Contact 
with a hard non-sticky wall anc capture by a detent also have been simulated convincingly by the 
manipulandum, as illustrated in Figures 3 and 4. Together, the performance capabilities described in 
this paragraph successfully meet a set of criteria used in the industry to demonstrate that an “artificial 
control feel” device is versatile enough to faithfully execute any general simulation (Jex 1988). 


REFERENCE 


H. R. Jex. Four critical test for control-feel simulators. Proceedings of the 23rd Conference on 
Manual Control, Cambridge MA, June 1988. 
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Figure 1. The manipulandum. Height from floor to spherical hand grip is 1 meter. 
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Figure 2. Virtual environment system components and communication. The highlighted frequencies 
{clockwise from upper left ) indicate the computer update rates for the visual display and load 
controller, and the manipulandum force bandwidth as determined by its first structural mode. 
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Figure 3. Wall model and implementations. The wall is modelled (left ) as a massless plate backed 
by a stiff spring and viscous damper. When the handle is to the right of the wall rest position (top ), 
no resistance is encountered. As the handle moves past the wall rest position (bottom ), the spring 
and dampers oppose further motion to the left. Torques resulting from the hand force applied to the 
constant length manipulandum handle are plotted against displacement ( right ) to demonstrate the 
effect of bandwidth on wall quality. The upper plot shows wall performance with a 20 msec lag 
introduced prior to detection of the handle position by the computer. The lower plot has this lag 
removed. Both simulations are updated at 1kHz. The lag causes a perceived “stickiness,” visible in 
the upper plot as the attractive force pulling the handle back toward the wall (i.e., displacement = 0). 
Away from the wall, the difference between mean force levels during approach and withdrawal 
motions is due to uncompensated friction. 
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Figure 4. Detent model and implementation. The detent is modelled ( top ) as a spring loaded ball 
plunger located at the bottom of the manipulandum handle that is trapped when it passes over the 
indentation in the planar workspace. The plot shows the history of torque due to handle force versus 
handle displacement in one of the manipulandum degrees of freedom as the handle is captured by a 
simulated detent, comes to a stop, and then breaks free. 
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Head-mounted, virtual world technologies were developed in the 1960s. Comeau and Biyan 
(1961) reported on a head-slaved video control system. In 1968, Sutherland discussed a head-slaved, 
graphics based virtual world that was later demonstrated by Vickers (1970). Operational systems for 
military aircraft were quickly developed and tested (Johnson and Foster, 1977). 

As part of second generation technology efforts, new display media and computer graphics sys- 
tems have been developed (Fisher, McGreevey, Humphries, and Robinett, 1986). With this new 
technology effort has come more systematic studies of the performance consequences resulting from 
conducting teleoperations in aeronautical settings. 

There are two primary purposes for head-mounted systems in aeronautical settings. One is for 
helmet-mounted sights and teleoperated (head-slaved) weapons systems. Bennett, Johnson, Perrone, 
apd Phatak (1988) evaluated head tracking performance during passive and controlled flight. In that 
study, comparisons were also made of head tracking performance in sterile and relatively complex 
virtual worlds. That study confirmed the robustness of head tracking performance across a wide 
variety of visual scenes. 

A second use of virtual world displays is for aircraft control. Aircraft controllability using head- 
mounted, panel-mounted or simulated out-the-window scenes has been systematically examined 
(Bennett, O’Donnell, and Johnson, 1988; O’Donnell, Johnson, and Bennett, 1988). Those studies 
reported the range of rotorcraft flight tasks in which head-mounted virtual worlds provided some 
advantages. 

Two studies will be reported that examine the perceptual/performance effects of virtual worlds. 
The first examines head tracking performance with roll-stabilized versus non-roll stabilized virtual 
worlds. The purpose of the study was to (a) examine display strategies used in current display sys- 
tems and (b) study the adaptability of observers to estimated glide slope angles using head-slaved 
versus head-stabilized imagery. The purpose of this study was to examine the usefulness of wide 
field-of-regards during final approaches to a runway. 
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The Stanford/VA Interactive Robotics Laboratory set out in 1978 to test the hypothesis that 
industrial robotics technology could be applied to serve the manipulation needs of severely impaired 
individuals. Five generations of hardware, three generations of system software, and over 125 exper- 
imental subjects later, we believe that genuine utility is achievable. The experience includes devel- 
opment of over 65 task applications using voiced command, joystick control, natural language com- 
mand and 3D object designation technology. 

A brief foray into virtual environments, using flight simulator technology, was instructive. If 
reality and virtuality come for comparable prices, you cannot beat reality. 

A detailed review of assistive robot anatomy and the performance specifications needed to 
achieve cost/beneficial utility will be used to support discussion of the future of rehabilitation teler- 
obotics. Poised on the threshold of commercial viability, but constrained by the high cost of techni- 
cally adequate manipulators, this worthy application domain flounders temporarily. In the long run, 
it will be the user interface that governs utility. 
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The use of the drug thalidomide by pregnant mothers in Britain resulted in a variety of deformi- 
ties including the birth of children having no arms. Such children were provided with powered artifi- 
cial arms with up to five degrees of freedom simultaneously controlled in real time by shoulder 
movement (ref. 1) and whose operation could be learnt by children within a matter of hours. The 
ease with which this manipulation could be learnt and used may have been due to the system used to 
provide feedback of position and force to the user’s skin and joints. In this way, the physiological 
sense of proprioception was extended from the user into the device, reducing the need for visual 
feedback and conscious control. 

With the banning of thalidomide, this technique fell into disuse but it is now being re-examined 
as a control mechanism for other artificial limbs (refs. 1-5) and it may have other medical 
applications to allow patients to control formerly paralysed limbs moved by electrical stimulation 
(ref. 6). It may also have commercial applications in robotic manipulation or physical interaction 
with virtual environments. 

c 

To allow it to be investigated further, the original pneumatic control system has recently been 
converted to an electrical analogue to allow interfacing to electronic and computer-assisted systems. 
A harness incorporates force-sensitive resistors and linear potentiomenters for sensing position and 
force at the interface with the skin, and miniature electric motors and lead screws for feeding back to 
the user the position of the robotic arm and the forces applied to it. In the present system, control is 
applied to four degrees of freedom using elevation/depression and protraction/retraction of each 
shoulder so that each collar-bone emulates a joystick. However, both electrical and mechanical 
components have been built in modular form to allow rapid replication and testing of a variety of 
force and position control strategies. 
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Rotation by 108° of the spatial mapping between a visual display and a manual input device pro- 
duces large spatial errors in a discrete aiming task. These errors are not easily corrected by voluntary 
mental effort, but the central nervous system does adapt gradually to the new mapping. Bemotat 
(1970) showed that adding true hand position to a 90° rotated display improved performance of a 
compensatory tracking task, but tracking error rose again upon removal of the explicit cue. This sug- 
gests that the explicit error signal did not induce changes in the neural mapping, but rather allowed 
the operator to reduce tracking error using a higher mental strategy. 

In this report, we describe an explicit visual display enhancement applied to a 108° -rotated dis- 
crete aiming task. A “wind indicator” corresponding to the effect of the mapping rotation is dis- 
played on the operator-controlled cursor. The human operator is instructed to oppose the virtual 
force represented by the indicator, as one would do if flying an airplane in a crosswind. This 
enhancement reduces spatial aiming error in the first 10 minutes of practice by an average of 70% 
when compared to a no enhancement control condition. Moreover, it produces adaptation aftereffect, 
which is evidence of learning by neural adaptation rather than by mental strategy. Finally, aiming 
error does not rise upon removal of the explicit cue. 


REFERENCE 


Bemotat, R.K. (1970) Rotation of visual reference systems and its influence on control quality. IEEE 
Transactions on Man-Machine Systems, vol. MMS-11, 129-131. 


This research was supported by Air Force Office of Scientific Research Grant #84-03-08, by National Aeronautics & 
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Cunningham. 
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Situation awareness, a current mental mode of the environment, is critical to the ability of opera- 
tors to perform complex and dynamic tasks. This should be particularly true for teleoperators, who 
are separated from the situation” they need to be aware of. The design of the man-machine interface 
must be guided by the goal of maintaining and enhancing situation awareness. 


The objective of this work has been to build a foundation upon which research in the area can 
proceed. A model of dynamic human decision making which is inclusive of situation awareness will 
be presented, along with a definition of situation awareness. A method for measuring situation 
awareness will also be presented as a tool for evaluating design concepts. The Situation Awareness 
Global Assessment Technique (SAGAT) is an objective measure of situation awareness originally 
developed for the fighter cockpit environment. The results of SAGAT validation efforts will be pre- 
sented. Implications of this research for teleoperators and other operators of dynamic systems will be 
discussed. 
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Virtual environments for software visualization may enable complex programs to be created and 
maintained. A typical application might be for control of regional electric power systems. As these 
encompass broader computer networks than ever, construction of such systems becomes very diffi- 
cult. Conventional text-oriented environments are useful in programming individual processors. 
However, they are obviously insufficient to program a large and complicated system, that includes 
large numbers of computers connected to each other; such programming is called “programming in 
the large.” 

As a solution for this problem, the authors are developing a graphic programming environment 
wherein one can visualize complicated software in virtual 3D world. One of the major features of the 
environment is the 3D representation of concurrent process. 3D representation is used to supply both 
network-wide interprocess programming capability (capability for “programming in the large”) and 
real-time programming capability. The authors’ idea is to fuse both the block diagram (which is use- 
ful to check relationship among large number of processes or processors) and the time chart (which 
is useful to check precise timin g for synchronization) into a single 3D space. The 3D representation 
gives us a capability for direct and intuitive p lanning or understanding of complicated relationship 
among many concurrent processes. 

To realize the 3D representation, a technology to enable easy handling of virtual 3D object is a 
definite necessity. Using a stereo display system and a gesture input device (VPL DataGlove), our 
prototype of the virtual workstation has been implemented. The workstation can supply the 
“sensation” of the virtual 3D space to a programmer. Software for the 3D programming environment 
is implemented on the workstation. According to preliminary assessments, a 50% reduction of pro- 
gramming effort is achieved by using the virtual 3D environment. The authors expect that the 3D 
environment has considerable potential in the field of software engineering. 

PROGRAMMING IN THE LARGE 

The authors are involved in the design project for future power supply system in Tokyo. The goal 
of the project is to have the entire power distributing system controlled automatically by a vast com- 
puter networking system. In order to realize that goal, we need to find ways to solve the problems 
which arise in developing such a large and complicated software scheme. 

Two basic types of problems associated with the different levels of the networking environment 
can be considered. One type of question might be: “How do I program each computer?” (This is 
called programming in the small). Another mights be: “How do I coordinate many computers?.” 
(This is called programming in the large.) 
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Since the programming environment itself is built on “algorithmic” or “logical” characteristics, it 
is convenient to use text representation. With text, we can eliminate much ambiguity. However, text 
alone becomes limitingly insufficient in the case of “programming in the large” where the most 
important factor in the field concerns total configuration design rather than detailed logic design. 

Thus a more global methodology is needed to correct for faults in the overall system design, even 
though each individual subsystem may be correctly designed. 

In “programming in the large,” the authors believe that visual representations play an important 
role in die intuitive understanding of software and a combination of text and visual environments 
will provide the best solution. 

CONCEPT OF 3D VISUAL REPRESENTATION 

Often, a network diagram is given as a 2D representation, known as block diagram. However, if 
the description of a real-time control program is required, such as in the synchronization of several 
control processes, the time dimension should also be taken into consideration. 

Fig. 1 shows the concept of the 3D visual representation. If we observe the representation along 
the X axis, it will be a conventional block diagram. The axis normal to the X axis is a time chart. 
Using this 3D representation, the programmer can intuitively grasp the state of concurrent process. 

In the 3D visual representation, the geometrical shape used to represent program flow of succes- 
sive message passing is also important. The shape itself indicates the type of information exchange. 
Sometimes the programmer can handle the software entirely based on its shape. The greater the 
number of processes, the greater the advantages of 3D representation, presumably due to characteris- 
tics of the human cognitive processes. 

IMPLEMENTATION 

Several years ago, such an idea was unrealizable due to the limitations of computer capabilities. 
To implement 3D representations, we need a new technology able to incorporates real-time 3D ani- 
mation, stereo displays, 3D input devices, and other aspects of the virtual 3D work-space. To gener- 
ate a 3D image, the authors used Stereographies CrystalEyes System, (a flickerless 3D CRT by high 
speed LC (120 Hz Liquid Crystal) shutter). By adding a head movement detector (Polhemus 3D 
tracker) to that system, a partial “look-around” effect can be implemented. As a gestural input 
device, VPL DataGlove is used. 

Fig. 2 shows the current hardware configuration of the virtual 3D workstation. The system 
includes two sub-workstations. One of them (an HP9000 SRX) is used for generating the virtual 3D 
image and handling I/O devices. Another ( a Sun 3) is used for text handling. In other words, the first 
one is for “programming in the large,” and the second is for “programming in the small.” 

Figure 3 shows an example of the visual programming environment. The message flow between 
processes is displayed in virtual 3D space. One small box represents a computing object (sometimes 
it can be a process, sometimes it can be a processor). 
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Semi-tranparent box represents a carrier of computing objects (it can be a stand-alone computer 
box). 


Using a virtual hand which works as 30 pointer and a kind of “action menu” which detects ges- 
tures, programmer can handle virtual objects (such as “objects” and “messages”) in the virtual 3D 
world. Currently, the environment has two modes, called “world mode” and “succession mode.” 
World mode is mainly for observing whole shapes of the software. Succession mode is used for edit- 
ing messages passing among objects. The configuration is still preliminary. Further investigation is 
needed concerning the design of object handling in 3D virtual space. 

Tools included in the environment are as follows: 

1 . Virtual Measure/Ruler: Tool to measure the exact relationship among virtual objects. 

2. Critical Path Finder: Tool to find and display the critical path which determines the total 
network throughput for a given network task. 

3. Network Simulator. Tool to simulate message passing and data processing on each computer. 

4. Network Planner: Tool to update information about the network configuration. This tool 
works with Network Simulator. 

ASSESSMENTS AND CONCLUSION 


As a preliminary assessment of the 3D environment, several multiprocessor programming tasks 
were assigned to subjects. The target system is supposed to be 3-4 networked personal computers. 
Completion time with and without the environment was measured. Roughly speaking, completion 
time with the 3D environment was shorter than that of the conventional methodology. In addition, 
using 3D lowers the variance, indicating that the environment reduces error. 

The authors believe that the concept of 3D visual representations has proven to be effective even 
in the simple environment already implemented. A far greater impact can be expected in a more 
sophisticated environment as the most case of software environment. For example, Smalltalk80 is 
highly valued for its concept of “object oriented” programming, but its environment is valued much 
more than the concept itself. The authors consider this kind of practical effort is indispensable. From 
a practical point of view, however, much effort is needed to prepare and accumulate effective tools 
for this new environment. 
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Figure 3. Example of visual 3D programming environment. 
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Vibrotactile displays have been designed and used as a sensory aid for the blind. In the present 
work the same 6 x 24 “Optacon” type vibrotactile display (VTD) was used to characterize human 
operator (HO) tracking performance in pursuit mid compensatory tasks. The VTD was connected via 
a microprocessor to a one-dimensional joy stick manipulator. Various display schemes were tested 
on the VDT, and were also compared to visual tracking performance using a specially constructed 
photo diode matrix display comparable to the VTD. 


Optimal performance, measured as minimal rms error between input and output signals, was 
achieved by dividing the VDT matrix into two 3 x 24 fields. On one side,l the reference signal was 
presented and on the second half the target position. The axial position on either half was generated 
by cycling between two adjacent 3 pin lines at 40Hz, moving up and down with the target, or refer- 
ence signals. Pseudo random bandlimited signals, DC to 0.4HZ were used testing the performance of 
four subjects. 


Results: 


1 . Very little training was needed by the subjects to learn to use the system and there was little 
improvement with time after the quick initial learning phase. 

2. No significant differences were measured between pursuit and compensatory tasks, in 
contrast to the visual tracking experiment. 

3. The calculated coherence function for both tasks was close to 1 .0 at the above dynamic 
range. 

4. The HO transfer function, after the exclusion of a fixed time delay, was a constant gain and 
constant phase at the above range. 

5. An average of about 420 msec, time delay between the input and output signals was 
exhibited by three subjects while the fourth subject had almost double this delay. 
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IMPLEMENTATION ISSUES FOR A COMPACT 6 DEGREE OF FREEDOM 
FORCE REFLECTING HANDCONTROLLER 
WITH CUEING OF MODES 


Heidi Jacobus, Alan J. Riggs, Charles Jacobus, and Yechiel Weinstein 
Cybernet Systems Corporation 
Ann Arbor, Michigan 


Teleoperated control requires a master human interface device that can provide haptic input and 
output which reflects the responses of a slave robotic system. The effort reported in this paper 
addresses the design and prototyping of a six degree-of-freedom (DOF) Cartesian coordinate hand 
controller for this purpose. The device design recommended is an XYZ stage attached to a three-roll 
wrist which positions a flight-type handgrip. Six degrees of freedom are transduced and control 
brushless DC motor servo electronics similar in design to those used in computer controlled robotic 
manipulators. This general approach supports scaled force, velocity, and position feedback to aid an 
operator in achieving telepresence. The generality of the device and control system characteristics 
allow the use of inverse dynamics robotic control methodology to project slave robot system forces 
and inertias to the operator (in scaled form) and at the same time to reduce the apparent inertia of the 
robotic handcontroller itself. 


The handcontroller unit which was designed and simulated in the initial effort uses brushless DC 
servo motors, integral pancake resolvers, and limited gearing to produce an extremely simple and 
rugged mechanical design requiring minimal routine maintenance. The electronic control unit 
designed is a compact low power robotic driver which supports robotic manipulator systems needing 
up to nominally 1500 W of power from a personal computer sized package. The current control 
design, which is not multiple fault tolerant, can be extended in this area to make flight control or 
space use possible. 


The proposed handcontroller will have advantages in space-based applications where an 
operator must control several robot arms in a simultaneous and coordinated fashion. It will also have 
applications in intravehicular activities (within the Space Station) such as microgravity experiments 
in metallurgy and biological experiments that require isolation from the astronauts’ environment. For 
ground applications, the handcontroller will be useful in underwater activities where the generality 
of the proposed handcontroller becomes an asset for operation of many different manipulator types. 
Also applications will emerge in the Military, Construction, and Maintenance/Manufacturing 
areas including ordnance handling, mine removal, NBC (Nuclear, Chemical, Biological) operations, 
control of vehicles, and operating strength and agility enhanced machines. Future avionics 
applications including advanced helicopter and aircraft control may also become important. 


1. The work reported here has been supported by NASA Johnson Space Center under contract NAS-9-18094. 

2. Poster version. 
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A human’s ability to perform physical tasks is limited, not only by his 1 intelligence, but by his 
physical strength. If, in an appropriate environment, a machine’s mechanical power is closely inte- 
grated with a human arm’s mechanical power under the control of the human intellect, the resulting 
system will be superior to a loosely integrated combination of a human and a fully automated robot. 
Therefore, we must develop a fundamental solution to the problem of “extending” human mechani- 
cal power. The work presented here defines “extenders” as a class of robot manipulators worn by 
humans to increase human mechanical strength, while the wearer’s intellect remains the central con- 
trol system for manipulating the extender. The human, in physical contact with the extender, 
exchanges power and information signals with the extender. 


The aim is to determine the fundamental building blocks of an intelligent controller, a controller 
which allows interaction between humans and a broad class of computer-controlled machines via 
simultaneous exchange of both power and information signals. Traditionally, human interaction 
with active systems (self-powered machines) has been defined as the exchange of “information 
signals” only. For example, the human sends information signals to an electric mixer (an active 
system) by pushing a “start” button, but the human does not transfer power to the mixer and does not 
feel the actual load on the mixer blades. As in this example, the prevalent trend in automation has 
been to physically separated the human from the machine so the human must always send 
information signals via an intermediary device (e.g., joystick, pushbutton, lightswitch). Extenders, 
however are perfect examples of self-powered machines that are built and controlled for the optimal 
exchange of power and information signals with humans. The human wearing the extender is in 
physical contact with the machine, so power transfer is unavoidable and information signals from the 
human help to control the machine. Commands are transferred to the extender via the contact forces 
and the EMG signals between the wearer and the extender, thus eliminating the need for a joystick, 
pushbutton, lightswitch or keyboard. This allows the person to maneuver his hand more naturally. If 
“talking” is defined as a natural method of communication between two people, then we would like 
to communicate with a computer naturally by talking rather than by using a keyboard. The same is 
true here: if we define “maneuvering the hand” as a natural method of moving loads, then we would 
like to move only the hand to maneuver a load, as opposed to using a keyboard or joystick. The 
extender augments human motor ability without accepting any explicit commands: it accepts the 
EMG signals and the contact force between the person’s arm and the extender, and the extender 
“translates” them into a desired position. In this unique configuration, mechanical power transfer 
between the human and the extender occurs because the human is pushing against the extender. The 
extender transfers to the human’s hand, in feedback fashion, a scaled-down version of the actual 


1 . The pronouns “he,” “his,” and “him” used in this abstract are not meant to be gender-specific. 
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external load which the extender is manipulating. This natural feedback force on the human’s hand 
allows him to “feel” a modified version of the external forces on the extender. 

Further clarification of the concept of transfer of power and information signals follows. The 
information signals from the human (e.g., EMG signals) to the computer reflect human cognitive 
ability, and the power transfer between the human and the machine (e.g., physical interaction) 
reflects human physical ability. Thus the information transfer to the machine augments cognitive 
ability, and the power transfer augments motor ability. These two actions are coupled through the 
human cognitive/motor dynamic behavior. The goal is to derive the control rules for a class of com- 
puter-controlled machines that augment human physical and cognitive abilities in certain manipula- 
tive tasks. 
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A TWO DIMENSIONAL ARTIFICIAL REALITY 

Myron W. Krueger 
Artificial Reality 
Vernon, Connecticut 


The current presumption is that is necessary to don goggles, gloves and a data suit to experience 
artificial reality. However, there is another technology that offers an alternative or complement to the 
encumbering techniques associated with NASA. In VIDEOPLACE, your image appears in a 2D 
graphic world created by a computer. The VIDEOPLACE computer can analyze a person’s image in 
1/30 second and can detect when an object is touched (Fig. 1). Thus, it can generate a graphic or 
auditory response to an illusory contact. 

VIDEOPLACE technology exists in two formats: the VIDEODESK and the VIDEOPLACE. In 
the VIDEODESK environment, the image of your hands can be used to perform the normal mouse 
functions, such as menuing and drawing (Fig. 2). In addition, you have the advantage of multippoint 
control. For instance, you can use the thumbs and forefingers of each hand as control points for a 
spline curve (Fig. 3). Perhaps most important, the image of your hands can be compressed and 
transmitted to a colleague over an ISDN voice channel to appear on the remote screen superimposed 
over identical information. Likewise, the image of your colleague’s hands can appear on both 
screens. The result is that the two of you can use your hands to point to features on your respective 
screens as you speak, exactly as you would if you were sitting together (Fig. 4). 

In the VIDEOPLACE environment, you can interact with graphic creatures and the images of 
other people in other locations in a graphic world. Your whole body can be moved, scaled and 
rotated in real-time without regard to the laws of physics. Thus, VIDEOPLACE can be used to create 
a fantasy world in which the laws of cause and effect are composed by an artist (Fig. 5) (Fig. 6). 
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Figure 1. Video touch. 



Figure 2. Videodesk drawing. 



Figure 3. Multipoint control. 
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Figure 5. Landing in an artificial reality. 



Figure 6. Parent and child size reversal. 
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BINAURAL ROOM SIMULATION 


H. Lehnert, J. Blauert, and W. Pompetzki 
Ruhr Universitat Bochum 
Bochum, FR GERMANY 


In every-day listening the auditory event perceived by a listener is determined not only by the 
sound signal that a sound emits but also by a variety of environmental parameters. These parameters 
are the position, orientation and directional characteristics of the sound source, the listener’s position 
and orientation, the geometrical and acoustical properties of surfaces which affect the sound field 
and the sound propagation properties of the surrounding fluid. A complete set of these parameters 
can be called an Acoustic Environment. If the auditory event perceived by a listener is manipulated 
in such a way that the listener is shifted acoustically into a different acoustic environment without 
moving himself physically, a Virtual Acoustic Environment has been created. Here, we deal with a 
special technique to set up nearly arbitrary Virtual Acoustic Environments, the Binaural Room 
Simulation. The purpose of the Binaural Room Simulation is to compute the binaural impulse 
response related to a virtual acoustic environment taking into account all parameters mentioned 
above. 

One possible way to describe a Virtual Acoustic Environment is the concept of the virtual sound 
sources. Each of the virtual sources emits a certain signal which is correlated but not necessarily 
identical with the signal emitted by the direct sound source. If source and receiver are non moving, 
the acoustic environment becomes a linear time-invariant system. Then, the Binaural Impulse 
Response from the source to a listener’s eardrums contains all relevant auditory information related 
to the Virtual Acoustic Environment. Listening into the simulated environment can easily be 
achieved by convolving the Binaural Impulse Response with dry signals and representing the results 
via headphones. 

Our simulation system contains the following steps. First the parameters of the environment to be 
simulated are entered into the computer using a CAD program geometric data of the room and 
databases for acoustic data of the walls and the directional characteristics. Position and orientation of 
source and receiver are defined by local coordinate systems. Then, the positions and orientations of 
the virtual sound sources are computed by ray tracing or mirror imaging algorithms (refs. 1, 2, 
and 3). The signals of the virtual sources are derived by convoluting all impulse responses of the 
walls involved in the sound path of the virtual source. The binaural room impulse response is then 
computed by propagating the signal of each virtual source to the listener and convolving the result 
with the external-ear impulse response of the listener for the respective direction of incidence. 
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We describe a virtual sound display built around a 12 MHz 80286 microcomputer and special- 
purpose analog hardware. The display implements most of the primary cues for sound localization in 
the ear-level plane. Static information about direction is conveyed by interaural time differences and, 
for frequencies above 1800 Hz, by head sound shadow (interaural intensity differences) and pinna 
sound shadow. Static information about distance is conveyed by variation in sound pressure (first 
power law) for all frequencies, by additional attenuation in the higher frequencies (simulating atmo- 
spheric absorption), and by the proportion of direct to reverberant sound. When the user actively 
locomotes, the changing angular position of the source occasioned by head rotations provides further 
information about direction and the changing angular velocity produced by head translations (motion 
parallax) provides further information about distance. Judging both from informal observations by 
users and from objective data obtained in an experiment on homing to virtual and real sounds, we 
conclude that simple displays such as this are effective in creating the perception of external sounds 
to which subjects can home with accuracy and ease. 
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In order to understand where human-machine interface hardware is headed, it is important to 
understand where we are today, how we got there, and what our goals for the future are. 




As computers become more capable, faster, and programs become more sophisticated, it 
becomes apparent that the interface hardware is the key to an exciting future in computing. One only 
has to look at CAD workstation hardware and the array of tablet, stylus, trackball, knob, and mouse 
devices which are combined with a maze of menus and function keys to understand the issues. These 
programs can create visual representations in 2 or 3-D, illustrate attributes of the object, such as 
stress and temperature distributions, or model and animate the object as it interacts with other 
objects. How can a user interact and control this seemingly limitless array of parameters effectively? 
Today, the answer is most often a limitless array of controls. The link between these controls and 
human sensory motor capabilities does not utilize existing human capabilities to their full extent. 


Imagine if the developers of the first car had electronic controls at their disposal and decided to 
control all car functions through a tablet or an array of switches or dials. They might have argued 
against a steering wheel as follows, “If a steering wheel is used, both hands are completely occupied 
at all time. How would you control the speed, or any of the other functions?” The rebuttal might 
have been, “Use foot pedals which is natural due to experience with walking or cycling,” The tablet 
advocate would have said, “No one will want to use their feet, it’s just not natural. Even if people 
would try it, it wouldn’t be smooth and it would damage the whole concept of the car.” As you can 
see, if cars evolved this way, they would be a maze of dials, menus and controls which saturate 
human sensory capabilities. We’d probably have speed limits of 30 mph due to the limitations in the 
reaction time as set by the interface hardware. Luckily, the car evolved to take more advantage of 
human capabilities. 


To use the car analogy again, when the car was bom, they didn’t ask what a car in the year 2000 
would look like and leave “hooks” and “scars” for improved technology. If they had, they would 
have been wrong, as few if any people are able to anticipate technology breakthroughs. They did, 
however, provide the best that was possible at the time and got the product out to as many people as 
possible and let time, experience, market demand, and technology innovation lead to the future. 

Interface hardware for teleoperation and virtual environments is now facing just such a crossroad 
in design. Therefore, we as developers need to explore how the combination of interface hardware, 
human capabilities, and user experience can be blended to get the best performance today and in the 
future. 
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In many movements of the upper limb such as reaching, positioning, and diplacing the objects, 
the hand moves smoothly along a uni-directional planar trajectory with a bell-shaped speed profile. 


Because of variability of the load, gravitational and velocity interaction forces between its seg- 
ments, the dynamics of the arm during the motion is very complex. Motion of the upper arm and the 
forearm are coordinated to produce smooth movements despite such complex dynamics. This coor- 
dination constitutes an important organizational feature of arm movement. The present paper 
describes some experimental results related to the above and interprets their role in producing 
smooth motion of the hand. 


Trajectories of the right upper limb in vertical plane and the simultaneous muscles’ EMG activi- 
ties were recorded for extensions of various amplitudes under four loads. The forearm trajectories 
are smooth with bell-shaped speed profiles. The upper arm trajectories may have bimodal speed 
profiles and three segments, resembling an inverted “Z.” Segmentation is sharper for the points near 
the shoulder joint, and is accentuated by load. As one moves from the central points near the shoul- 
der joint to the peripheral points near the hand, the three segments merge and result in smooth curves 
with single-peaked speed profiles. The three segments have a coherent time course and can be iden- 
tified rather accurately and non-ambiguously. 

The observed trajectories and the EMG patterns reveal an effective coordination strategy which 
utilizes the structure and the dynamics of the moving arm to produce smooth movement. 
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X. D. Pang, H. Z. Tan, and N. I. Durlach 
Massachusetts Institute of Technology 
Cambridge, Massachusetts 

Optimal design of human-machine interfaces for teleoperators and virtual-environment systems 
which involve the tactual and kinesthetic modalities requires knowledge of the human’s resolving 
power in these modalities. The resolution of the interface should be appropriately matched to that of 
the human operator. We report some preliminary results on the ability of the human hand to distin- 
guish small differences in force under a variety of conditions. Experiments were conducted on force 
discrimination with the thumb pushing an interface that exerts a constant force over the pushing dis- 
tance and the index finger pressing against a fixed support. The dependence of the sensitivity index 
d’ on force increment can be fit by a straight line through the origin and the just-noticeable differ- 
ence (JND) in force can thus be described by the inverse of the slope of this line. The receiver-oper- 
ating-characteristic (ROC) was measured by varying the a priori probabilities of the two alternatives, 
reference force and reference force plus an increment, in one-interval, two-alternative, forced-choice 
experiments. When plotted on normal deviate coordinates, the ROCs were roughly straight lines of 
unit slope, thus supporting the assumption of equal-variance normal distributions and the use of the 
conventional d’ measure. The JND was roughly 6-8% for reference force ranging from 2.5 to 
10 newtons, pushing distance from 5 to 30 mm, and initial finger-span from 45 to 125 mm. Also, the 
JND remained the same when the subjects were instructed to change the average speed of pushing 
from 23 to 153 mm/sec. The pushing was terminated by reaching either a wall or a well, and the 
JNDs were essentially the same in both cases. 
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The advent of high resolution, physical model based computer graphics has left a gap in the 
design of input/output technology appropriate for interacting with such complex virtual world mod- 
els. Since virtual worlds consist of physical models, it is appropriate to output the inherent force 
information necessary for the simulation to the user. The detailed design, construction, and control of 
a three degree freedom force output joystick will be presented. 

A novel kinematic design allows all three axes to be uncoupled, so that the system inertia matrix 
is diagonal. The two planar axes are actuated through an offset gimbal, and the third through a 
sleeved cable. To compensate for friction and inertia effects, this transmission is controlled by a 
force feedforward and a closed force feedback proportional loop. Workspace volume is a cone of 
512 cubic inches, and the device bandwidth is maximized at 60 Hz for the two planar and 30 Hz for 
the third axis. Each axis is controlled by a motor/proportional magnetic particle brake combination 
fixed to the base. The innovative use of motors and brakes allows objects with high resistive torque 
requirements to be simulated without the stability and related safety issues involved with high 
torque, energy storing motors alone. Position, velocity, and applied endpoint force are sensed 
directly. Different control strategies are discussed and implemented, with an emphasis on how vir- 
tual environment force information, generated by the MIT Media Lab Computer Graphics and Ani- 
mation Group BOLIO system, is transmitted to the device controller. 

The design constraints for a kinesthetic force feedback device can be summarized as: 

How can the symbiosis between the sense of presence in the virtual environment be maximized 
without compromising the interaction task, under the constraints of the mechanical device 
limitations? 

Research in this field will yield insights to the optimal human sensory feedback mix for a wide 
spectrum of control and interaction problems. A flexible research tool that is designed as an easily 
reproducible product prototype has been constructed to explore the variety of possible force 
interaction. 
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Introduction 


The advent of high resolution, physical model based computer graphics has left a gap in the 
design of input/output technology appropriate for interacting with such complex virtual world mod- 
els. Visual display technology has improved dramatically, driven by developing communication 
through the most information rich sensory channel. Advances in stereographic displays and more 
proprioceptive input devices that contain higher degreee of freedom position information, such as 
full hand or body gesture recognition, still rely only on the visual sense. The user operates in a world 
of relative paralasys, not able to feel, grasp, touch, smell, or poke the objects that she interacts with. 

Since virtual worlds are based on physical models, it is appropriate to output the inherent force 
information necessary for the simulation to the user. Before one commits large resources to the con- 
struction of such force output machines, the appropriate design constraints and problem definitions 
for simulated environment interactions must be addressed. The problem can be summarised as: 

How can the symbiosis between the sense of presence in the virtual environment be maximised 
without compromising the interaction task, under the constraints of the mechanical device 
limitations? 

Research in this field will yield insights to the optimal human sensory feedback mix for a wide 
spectrum of control and interaction problems: telepresence and supervisory control in remote 
manipulator tasks, advanced computer assisted engineering design tools, and surgical or flight simu- 
lators among many others. A flexible research tool to explore the variety of possible force interac- 
tions and is nevertheless designed as an easily reproducable product prototype has been constructed. 

Overview and Design Specification 

Traditional force output devices have concentrated en large, global movements, motivated by the 
scale of the teleoperated robot motion. In general, an analog slave sensor feedback signal servos the 
master joint motors. The geometric configuration and physical dimensions of the controlled manipu- 
lator and of the master are the same. In many cases, however, geometric equality is not necessary 
and can even hinder a particular successful task completion. Recently, in the flexible, unspecified 
tasks that are possible with virtual environments, the goal is to design the force output device so that 
its own dynamics are transparent and yet also can simulate as wide a frequency band of forces as 
possible. The range of motion used in interacting with a computer can be limited to a small surface 
or volume, one need only think of the small “mouse” interface device movement. The design of a 
force output device for virtual environments should maximize the device fidelity under the general 
constraints of size, cost, limited computation speed, and safety. These are the general constraints that 
drove the design of the presented force feedback interface. The design catagories are detailed below. 
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Design Categories: 


Engineering Development: 

Design Simplicity: 

Appearance 

Limited Size 

Robust 

Versatile 

Modular 

Expandable 

Safety 

Difficulty of implementation 
Technological base 
Cost 

Controllability: 

Interface Transparency: 

Can be counterbalanced (Electronically or Mechanically) Backdrivable 
High DOF 
Uncoupled axes 
Low degree of computation: 

Full State measurement 

Integrated into system w/o control/graphic display interference 
Limited control logic 
High Position Resolution 
Uncoupled axes 
Closed or open loop control: 

Full State measurement 
Stability: 

Stiff 

High natural frequency 
Human-Device Interaction: 

Secondary Function Control 

Non-fatiguing 

Safety 

Human arm limitations never exceeded 
Interface Transparency 
Variable force feedback ratio 

The described force output joystick incorporates linear but no rotational force output. Many 
manual tasks in which force information is of utmost importance and that are completed with the use 
of tools can be limited to three linear degrees of freedom. The metaphor used as a design guideline, 
which allows for a tractable problem within the design requirements, is a tool endpoint force simula- 
tor. To maintain design simplicity and limit the size without compromising device versatility, the 
joystick endpoint is limited to the three degrees of linear motion. It can be upgraded to include a 
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small three degrees of freedom torque output mechanism, so a full 6 degrees of freedom force and 
torque output could be realised. 

Any point contact can by modulated according to the characteristics imposed by the simulation. 
The joystick endpoint is a single probe with which to explore the virtual computer space. How 
effective such an approach is in transmitting tactile information is a matter of ongoing research 
(Minsky). Full, high resolution tactile displays have a large number of independent degrees of free- 
dom, determined by the size and resolution of the display. Each tactile “pixel” must be indepen- 
dently controlled, so devices of this kind are difficult to implement. Examples include Braille 
machines for the blind, which are the state of the art for such devices, and it is an open question if 
their resolution is sufficient for effective, more general representation tactile displays. 

A fine tool is not grasped with the whole hand, but rather like a pen between one’s fingers. The 
force magnitude in such and interaction is much lower than in full arm movement. The restriction to 
finer forces and motions, which is appropriate for the limited workspace requirement of the human 
arm and hand, allows the joystick to be constrained in size since the actuator brakes and motors can 
have lower torque ouputs. Also, the smaller motors more easily meet the safety requirements and the 
limitation to three degrees of freedom decreases the difficulty of implementation and cost. The three 
degrees of freedom can be chosen such that the axes are perpendicular and all the actuators are 
attached to ground, thus eliminating cross coupling between the endpoint principal directions and the 
actuator torques. Rotary, no torque, ripple, low inertia motors and proportional magnetic particle 
brakes with a simple cable transmission for the third axis, and most of the sensors are readily avail- 
able from existing technology. Only a three axis linear load cell of sufficiently small size and mass 
had to be specifically designed and constructed. 

Control Requirements 

The detailed mechanical design of the joystick is dominated by the control requirements of the 
simulation. To both impose the arbitrary simulation dynamics onto the joystick endpoint and mini- 
mize user fatigue when the joystick is used as a simple position input device, the actual characteristic 
or plant dynamics should not interfere with the simulation. This interface transparency is from the 
perspective of the user interacting with the joystick. One input to the plant is the force exerted by the 
user on the endpoint. The simulation, however, controls the desired dynamics through the input sig- 
nals to the motors and brakes. The control strategy needs to maintain the simulation dynamics under 
varying user force inputs, or under the changing coupling characteristics between the user’s arm and 
plant dynamics. The simulation essentially modulates the brake and motor signals so as to match the 
endpoint impedance or effective mass, stiffness, and damping characteristics to that of the virtual 
dynamics. The mechanical design maximizes the inherent device transparency and so minimized the 
control effort needed to backdrive the joystick. It has a bandwidty of 60 Hz for the gimbal driven 
axes and 30 Hz for the cable drive. Because of closed loop force control stability issues, open loop 
backdriveability is obtained wherever possible. This necessitates a direct drive between the brakes, 
motors, and joystick endpoint to minimize stiction, viscous friction and effective actuator inertia, 
which are all amplified by a transmission ratio. Direct drive in turn requires large motors for a signif- 
icant torque. All the motors and brakes are fixed to ground and thus their weight and reaction torques 
need not be compensated for. Because of the much higher torque output of the brake for a given size 
and power input, any high torque requirements of the simulation will be met by the brake, an 
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inherently passive, no energy storing and therefore safe torque device. Thus, both backdriveability 
and output force are maximized within the size and safety constraints. 

No known research has been conducted in the appropriate strategy for hybrid, independent pro- 
portional brake and motor control. This novel combination promises to yield new methods of 
increasing stability and safety bound in actively controlled mechanisms without decreasing the 
closed loop device stiffness. The control of brakes necessitates a direct measurement of the contact 
force between the endpoint and user. Both for this reason and to actively backdrive the R-axis 
transmission, a loadcell 3-axis force sensor is implemented in the device. 

The actual joystick endpoint position, velocity, and force are necessary to determine the charac- 
teristics of the interface dynamics. The servo loop sampling frequency is limited by the bandwidth of 
the simulated environment. As all the joystick control must operate in real time, the degree of com- 
putation lag in the control loop must be minimized. To achieve this and to eliminate quantization and 
noise errors at low velocities introduced by digital differentiation, both motor/brake shaft position 
and velocity are measured directly. The endpoint position and velocity can then be found through a 
simple matrix position and Jacobian transformation. The joystick linkage is chosen such that all 
actuator induced endpoint velocities are perpendicular. The axes are therefore uncoupled and any 
actuator exerts a force in one of the endpoint principle directions. The system inertia matrix is thus 
only diagonal, though time variant. 

Future Work 

Some simple texture and object simulation code has been implemented successfully. Present 
research efforts are concentrated on expanding the complexity of a real-time virtual world interaction 
by utilising the physically based graphics animation package in the MTT Media Lab Computer 
Graphics and Animation Group. Various studies of the effect of multiple sensory interaction on real- 
ism in simulations and the associated mechanical device complexity cost are proposed, with the pre- 
sent device as a research tool to quantify the effect of force information in successful task comple- 
tion and enhanced apparent realism. 
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Space Telerobotics research, performed under contract to the European Space Agency (ESA), 
concerning the execution of human factors experiments, and ultimately leading to the development 
of a telerobotics test bed, has been carried out since 1985 by a British Consortium consisting of 
British Aerospace, the United Kingdom Atomic Energy Authority and, more recently, the UK 
National Advanced Robotics Research Centre. The principal aim of the first study of the series was 
to derive preliminary requirements for a teleoperation servicing system, with reference to two mis- 
sion model scenarios. The first scenario introduced the problem of communications time delays, and 
their likely effect on the ground-based operator in control of a manipulator system on board an 
unmanned servicing vehicle in Low Earth Orbit. In the second scenario, the operator was located on 
the NASA Orbiter aft flight deck, supervising the control of a prototype manipulator in the 
“servicing” of an experimental payload in the cargo bay area. Human factors analyses centred on 
defining the requirements for the teleoperator workstation, such as identifying basic ergonomic 
requirements for workstation and panel layouts, defining teleoperation strategies, developing 
alphanumeric and graphic screen formats for. the supervision or direct control of the manipulator, and 
the potential applications of expert system technology. 

The second study for ESA involved an experimental appraisal of some of the important issues 
highlighted in the first study, for which relevant human factors data did not exist. Of central impor- 
tance during the second study was the issue of communications time delays and their effect on the 
manual control of a teleoperated manipulator from a ground-based command and control station. An 
experimental test bed, based around ASEA Irb6 and 60 robots was developed, incorporating a gen- 
eral purpose robot controller, a reconfigurable operator workstation and a modular task box. The 
programme was divided into a series of short pilot studies and two principal experiments. The aim of 
the pilot studies was to optimise the characteristics of hand controllers and visual feedback condi- 
tions prior to their more formal evaluation and comparison in the later experiments. The experiments 
were conducted using standard psychological paradigms, involving human performance comparisons 
over a range of 3 and 6 degree-of-freedom hand controllers, with TV and graphics displays in vari- 
ous configurations. Although a rigorous experimental approach was adopted during Teleoperation 
and Control Study 2, it was evident that this approach suffers a number of limitations. The most 
critical concerns the extent to which the data and conclusions from this study can be generalised to 
future scenarios. Also, it is recognised that the performance measures of time and accuracy alone 
were not satisfactory, as they each provided only an overall index of performance for a given trial, 
not an analytical index of performance within the trial. Clearly, more analytical techniques to analyse 
operator performance in detail are required, together with a methodological framework in which all 
techniques can be used to judge the effect of “cost” of a particular technology, not only on human 
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teleoperation performance, but also on the total telerobotic system, with the future aim of defining 
operational procedures. 

Teleoperation and Control Study 3 is investigating a new technique - console or performance 
“logging”- for the measurement of teleoperation performance to complement those measures used in 
the second study, such as primitive and whole-task timings, handling errors, debriefing and subjec- 
tive workload records (NASA TLX), video and audio recordings. Instrumenting the human-system 
interface to record and store both multiple gross actions and much finer keystroke-level parameters 
should provide a more detailed insight into die incidence, cause and effect of interactions between 
various control activities, and assist the experimenters in analyzing those control behaviours of too 
fine a nature to be extracted from video records alone (such as fine individual joystick motions). 

Such a keystroke level of analysis should provide data suitable for future performance modelling 
activities in the area of telerobotic command and control. 

Included among the parameters to be recorded in Study 3 are all robot control joystick axes, all 
additional joystick-mounted controls (where used), all panel-mounted switches, all camera joystick 
controls, all direct voice input utterances (incidence of onset only), all robot motions, as resolved at 
the end effector (i.e., translation and rotation), elapsed time (pulse code modulated), and experi- 
menter-induced event markers. Areas of interest for the analysis work include an appraisal of indi- 
vidual strategies and their relation to task success, across-condition comparisons of control 
characteristics (e.g., when stereo vision or predictor displays are used in the experimental designs), 
error history monitoring (i.e., analysis of “lead-up” events), analysis of discrete event changes (e.g., 
the effects of discrete time delay changes, or selections of new robot control frames), and possibly an 
appraisal of die effects of controller “physical” characteristics (e.g., axis cross-talk, deadband, break- 
out forces). Finally, there are a number of outstanding questions to be addressed with regard to the 
use and development of the data logging system described above. For instance, is it desirable and 
even possible to use the data to generate metrics of performance suitable for complementing experi- 
mental and subjective results? Do other forms of control data (other than onset, direction and dura- 
tion of control inputs) exist that are appropriate for logging? How can the concept be extended to 
record performance-related parameters other than control inputs (such as visual search, attention, 
mental workload, etc.)? Can a data logging system provide a useful insight into the operational cost 
of remote space operations? Answers to these questions will become clearer, following the use of the 
logging system during the experimental phase of the third contract, scheduled for the Summer of 
1990. 
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The National Advanced Robotics Centre was founded in June 1988, by the United Kingdom’s 
Department of Trade and Industry and 1 1 industrial and academic shareholding organisations. The 
Centre acts as a focal point for the development of the core technologies which facilitate the next 
generation of robots, and is geared toward projecting the UK to the front of this international arena. 
One of the key areas of the Centre’s enabling technologies research programme is that of the human- 
system interface, phase 1 of which started in July 1989 and is currently addressing the potential of 
virtual environments to permit intuitive and natural interactions between a human operator and a 
remote robotic vehicle. The aim of the first 12 months of this programme (to September, 1990) is to 
develop a virtual human-interface demonstrator for use later as a test bed for human factors experi- 
mentation. This presentation will describe the current state of development of the test bed, and will 
outline some human factors issues and problems for more general discussion. 


In brief, the virtual telepresence system for remote driving has been designed to take the follow- 
ing form.. The human operator will be provided with a helmet-mounted stereo display assembly, 
facilities for speech recognition and synthesis (using the Marconi Macrospeak system), and a VPL 
DataGlove Model 2 unit. The vehicle to be used for the purposes of remote driving is a Cybermotion 
Navmaster K2A system, which will be equipped with a stereo camera and microphone pair, mounted 
on a motorised high-speed pan-and-tilt head incorporating a closed-loop laser ranging sensor for 
camera convergence control (currently under contractual development). It will be possible to relay 
information to and from the vehicle and sensory system via an umbilical or RF link. The aim is to 
develop an interactive audio-visual display system capable of presenting combined stereo TV pic- 
tures and virtual graphics windows, the latter featuring control representations appropriate for vehi- 
cle driving and interaction using a graphical “hand,” slaved to the flex and tracking sensors of the 
DataGlove and an additional helmet-mounted Polhemus IsoTrack sensor. 


Developments planned for the virtual environment test bed include transfer of operator control 
between remote driving and remote manipulation, dextrous end effector integration, virtual force and 
tactile sensing (also the focus of a current ARRL contract, initially employing a 14-pneumatic blad- 
der glove attachment), and sensor-driven world modelling for total virtual environment generation 
and operator-assistance in remote scene interrogation. 


145 

23 








N95- 14061 


DESTABILIZING EFFECTS OF VISUAL ENVIRONMENT MOTIONS 
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Teleoperators are the humans who control devices from a “distance.” This distance might be 
extreme physical separation, as in remote assembly of Space Station Freedom by ground-based per- 
sonnel. But the distance could be effective distance of scale, for the manipulation of microscopic 
structures like single living cells or the components within an integrated circuit chip, or even the 
conceptual distance of safety, where the devices effect their actions inside a nuclear reactor’s core or 
in the ocean’s depths. Virtual environments refer to the synthesized realities that can be generated by 
various types of computerized displays, not only visual displays but also acoustic, tactile, and force- 
reflective displays. Video games are common but usually limited examples of virtual worlds. An 
implicit theme is telepresence, a term whose definition is imprecise. It involves the use of virtual 
environments to improve the efficiency of teleoperators by giving them a compelling sense of “being 
where the action is.” Telepresence operations seem more natural and facile, and thus more easily 
trained, than.other possible models for the human-computer interface. Consider a complex assembly 
task,. controlled by wearing a sensor garment that enslaves a robotic arm to mimic the user’s own 
arm movements, and visualized on video and felt by tactile and force-reflective feedbacks; versus, its 
control by typing code into a keyboard with feedback via numerical tables. Telepresence seems to 
endow the user with such a robust mental model that he or she becomes absorbed into the synthetic 
reality as though absorbed into a vivid dream. 

The primary challenge for reduction to practice is to develop a paradigm for human-computer 
interaction thatwill enable telepresence to be implemented reliably . The natural movements of the 
human operator, such as the ability to “look around” in the virtual world, provide an important com- 
ponent for this paradigm inasmuch as the consistency of the virtual world can be enhanced by its 
enslavement to sensed head or eye movements of the human user. 

In the present paper, we explore effects on the human of exposure to a visual virtual 
environment which has been enslaved to simulate the human user’ s head movements or eye 
movements. Specifically, we have studied the capacity of our experimental subjects to maintain 
stable spatial orientation in the context of moving their entire visible surroundings by using the 
parameters of the subjects’ natural movements. Our index of the subjects’ spatial orientation was the 
extent of involuntary sways of the body while attempting to stand still, as measured by translations 
and rotations of the head. We also observed, informally, their symptoms of motion sickness. 


^ow at Agricultural Research Service, U.S. Department of Agriculture. 
2 Now at Honeywell Phoenix Techology Center. 
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METHODS 


Subjects 

A total of 93 university students or staff, 50 males and 43 females ranging in age from 18 to 
43 years old, participated as unpaid volunteers in one of four experiments. Some students received 
course credit. Subjects were prescreened for medical histories (self-report) and any with possible 
vestibular defects were excused. Any subjects requiring refractive correction used their own pre- 
scribed lenses. 

Apparatus and Procedures 

Head Position Sensing. An acoustic timing principle was used to locate in space the headgear 
worn by each subject. This lightweight headgear contained two click sources, the wavefronts from 
which were detected with microsec timing resolution at each of four microphones attached to the 
ceiling of the testing room. Software algorithms for windowing the microphone outputs and per- 
forming redundancy checks, then implementing solutions of the Pythagorean theorem, permitted 
each click source to be located in space with translation precision below 1mm at a sampling rate of 
50 Hz maximum. Resolution of the rotation angle for the line connecting the two click sources was 
below 0.2 deg (25 Hz maximum sampling rate). The headgear configuration did not permit resolu- 
tion of rotations around the line connecting the click sources (pitch), but did permit resolution of the 
other five degrees of freedom for head movements. 

Visual Environment. Each subject stood near the axis of a vertical hemicylindrical screen onto 
which was projected a pattern of vertical stripes. This pattern subtended in excess of 180 deg of 
azimuth and 120 deg in elevation. The instantaneous azimuth positioning of the pattern was under 
software control with a resolution of approximately 0.04 deg at a maximum update rate exceeding 
lKHz. Impedances within the projection system were compensated for via nonlinear control signals 
when motions simulating saccadic eye movements were employed. In those particular experiments, 
the control signals were precomputed to reduce lag time to below 1 msec. When motions enslaved to 
the subject’s head movements were employed, it was impractical to precompute the control signals. 
Thus, there was an additional lag time for computation of about 150 msec and the update rate was 
reduced to 10 Hz. 

Simulations of Saccadic Eye Movements. Two small fixation lights rear-projected onto the sur- 
rounding screen could be alternated to direct horizontal saceades of particular azimuth extent (0.5, 1, 
2, 3, 4, 6, 8, or 16 deg). Motions of the pattern front-projected onto the screen had characteristics of 
average velocity and motion duration similar to those for each corresponding saccade extent. 3 In two 
experiments, the saccade-like stimulus motions occurred independent of the subject’s eye move- 
ments. In another experiment, the stimulus motions were triggered by subject eye velocity, derived 
from the electro-oculogram, when the eye had moved 15-20 arcmin from its initial fixation during a 
guided voluntary saccade. 


3 A. T. Bahill, M. R. Clark & L. Stark, Mathematical Biosciences, 24, 191-204 (1975). 

Partially supported by a grant from the National Eye Institute to KDW (R23 EY03640), and by an NSF graduate 
fellowship to JHK. 
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Head Movement Feedback Conditions. Each sample of the head’s fore/aft and left/right 
translations plus head yaw angle permitted real time calculation of an equivalent azimuth change for 
the surrounding visual stimulus. This equivalent azimuth, modified by a gain factor that was the 
parameter selected for study, controlled motion of the visual stimulus in this experiment. The gain 
factors used were +2, +1.5, +1, +0.5, 0, -0.5, -1, -1.5, and -2. The gain of +1, for example, could 
serve in principle to keep the stimulus “straight ahead” by its faithfully tracking the head’s 
movements. Larger gains moved the stimulus farther and faster than would accurate head tracking; 
smaller gains moved the stimulus less far and more slowly. Zero gain made the stimulus remain still 
as in the natural environment. Negative gains reversed the left/right correspondence between head 
movements and stimulus motions. 

Data Analysis 

Control Conditions. Although the stability of a standing human has been modelled as an 
inverted pendulum, since the body’s center of gravity is above its support base, the problem with 
such a model is that the human body is not mechanically rigid. Rather, it is a jointed mechanism 
controlled by over 400 muscles and several sensory feedback loops, with correspondingly complex 
dynamics. Furthermore, people grow to various heights, weights and distributions of body mass; 
with various extent of conditioning for their musculature; and with varied skills of sensorimotor 
integration. Individual variations in the body’s dynamics are, therefore, prominent and, as a 
consequence, we rely on the use of experimental designs allowing each individual to serve as his/her 
own control. 

Also, it is impractical to open the feedback loops from proprioceptive/kinesthetic or vestibular 
senses but it is relatively easy to alter those loops by asking the subject to stand only on one foot or 
else on a compliant surface. As reported in the results below, there are multiple techniques for con- 
trolling the visual feedback loop within each subject, including: (a) injecting noise via random 
motions of the visual environment, (b) stabilizing the environment via motion feedback from head 
movements, (c) eliminating contour from the visual environment, or (d) eliminating visibility by 
turning off the lights and (e) by closing the eyes. 1 

Within-Subject Figure of Merit for Body Stability. There is no consensus in the literature for a 
representative figure of merit to describe such complicated behavioral dynamics as those under con- 
sideration. We have not been satisfied, in general, with time-domain analyses such as average 
movement velocity or RMS position enror, since these statistical summaries tend to obscure details 
of the responses which we find to be correlated with experimental manipulations. We have, however, 
found one particular form of time-domain analysis to be useful in certain circumstances, and that 
form is time-locked response averaging. We plan to explore in future work the more general utility 
of a “chaotic” time-domain (phase-sensitive) analysis. 

We have enjoyed considerable success in accounting for individual variations among subjects by 
using frequency-domain analysis, however. Specifically, each subject is measured: (a) under control 
conditions to determine his/her individual power spectral density (PSD) functions for body move- 
ments over time, and (b) under experimental conditions, to determine any changes in the PSD 
functions induced experimentally. The ratio between PSD functions (a) and (b) is the gain, as a 
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function of the frequency, induced by the experimental manipulation. We find these gains to be 
relatively consistent betwen individuals (standard errors of measurement typically less than 1 dB). 


RESULTS 


Figure 1 shows sample records of two subjects’ lateral head movements while viewing moving 
surroundings (middle two traces). For the top trace, the motion of the stimulus was saccade-like, 
while for the bottom trace, the stimulus tracked the subject’s head movements (gain = +1.5). These 
sample records have been displaced vertically for clarity. 

The top trace of Figure 1 shows typical lateral sways made by subject S 1 when the entire visible 
surroundings suddenly moved with the velocity (66 deg/sec) and duration (30 msec) characteristic of 
a 2 deg saccadic eye movement. Note the relatively pronounced fluctuations in head position. By a 
comparison against the second trace, S l’s lateral sways in still surroundings, it is clear that the stim- 
ulus motion exerts a strong destabilizing effect. 

The bottom trace of Figure 1 shows lateral sways made by subject S 2 while the azimuth position 
of the visible stimulus tracked S 2’s own head movements. By comparison against the third trace, 
subject S 2’s lateral sways in still surroundings, the destabilizing effect of head tracking is apparent. 
Individual variations in stability can be noted by comparing the third trace (S 2) against the second 
trace (S 1), sways made in still surroundings. 

Figure 2 depicts averaged yaw movements (i.e., head rotations) made in reponse to saccade-like 
motions of the stimulus, plotted as a function of time after the motion began. Each plotted point is 
based on 10 repetitions per subject times 18 subjects, or the average of 180 measures of head yaw 
angle per condition. The measures were carried out in three conditions: (1) saccade-only, the subject 



Figure 1. Sample records of head movements in still vs. moving environments (saccade-like or head 
tracking motions), for two subjects. 


151 





Figure 2. Response averages showing head yaw elicited by saccade-like stimulus motions. Motions 
triggered by saccades are less effective than untriggered motions for eliciting these responses. 

moved their eyes as directed by fixation lights but there was no additional motion of the stimulus, 

(2) motion triggered by the saccade, in which a saccade-like stimulus motion took place while the 
subject’s eyes were moving in a saccade, and (3) untriggered motion, in which a saccade-like 
stimulus motion briefly preceded (<100 msec) the subject’s execution of a saccadic eye movement. 

In the saccade-only condition, subjects showed a slight tendency to yaw the head as well as to move 
the eyes. Average head yaw for saccade-only is taken as the baseline in this plot. This baseline 
motion is subtracted from the head yaws in response to stimulus motions. 

It is clear in Figure 2 that untriggered saccade-like stimulus motions elicit a small but reliable 
yaw (the peak at 0.5 sec represents 10 standard errors difference). This may be qualitatively similar 
to the lateral head translations illustrated in the top trace of Figure 1. However, Figure 2 also shows 
that the effect of the saccade-like stimulus motions can be lessened dramatically by causing them to 
coincide with the subject’s own voluntary saccades. In this latter case, possibly a signal generated 
internally by the subject’s oculomotor system (viz., corollary discharge or efference copy) alters the 
consequences that otherwise accompany the stimulus motion. 

Figure 3 is a plot derived from fast Fourier transforms of sway records analogous to those shown 
in Figure 1. For the plot shown here, the power spectral density function (PSD) for each subject was 
obtained in still surroundings and this was taken as the baseline for that individual. PSDs for that 
subject under other conditions of stimulation were divided by his/her own baseline PSD. This 
resulted in gain profiles (dB relative to baseline) as a function of sway frequency (Hz). Each point on 
Figure 3 represents the average gain across 24 subjects (18 for saccades condition) within a fre- 
quency band 0.5 Hz wide. Typical standard errors associated with each point are under 1 dB. 

The six curves shown in Figure 3 depict the effects on subjects’ stability of (1) closing their eyes, 
(2) eliminating visible contour (subjects wore diffusing goggles), (3) eliminating illumination 
(subjects wore opaque goggles, (4) moving the azimuth of the surroundings to track the subject’s 
head movements (gain ±1.0), (5) moving the azimuth of the surroundings randomly, and (6) the 
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Figure 3. Comparisons of the gain in lateral sway as a function of sway frequency, for experimental 
and control condition. 

subjects made directed saccadic eye movements across the still surroundings. Two of the curves on 
Figure 3 were moved vertically to reduce overlap: the “lights out” curve was moved down 2 dB and 
the “head track” curve was moved down 3 dB. “Eyes closed” caused significantly larger sways than 
either “no contours” or “lights out,” which did not differ reliably. “Head track” resulted in signifi- 
cantly smaller sways than in those conditions. 

All six curves show a general trend of decreasing gain with increasing frequency, particularly so 
below 2 Hz. To a first approximation, the gain profile associated with any of these six conditions 
appears nearly interchangeable with the profile from another condition by simply shifting the curve 
up or down on the graph. Since all of these conditions might serve to reduce or degrade visual feed- 
back from the environment, this first approximation similarity is not overly surprising. In detail, 
however, it is interesting to note that the saccades condition (6) actually made die subjects sway less 
than did fixating in still surroundings. Also note that the head track condition (4) is the only curve 
with a large gain decrease at 0.75 Hz (midpoint for the 0.5 to 1.0 Hz band). The frequency depen- 
dence of gain profiles with head tracking is more apparent in the following graphs. 

Figures 4 and 5 compare the effects of four different gain factors for head tracking. The gain 
factor parameter relates azimuth positioning of the visible surround to the combined fore-aft and left- 
right translations plus yaw angle sensed for the head. Each point on these figures represents the aver- 
age movement gain across 33 subjects, calculated within 0.5 Hz bandwidths. These gains, the ratios 
of PSD profiles in tracking vs. baseline conditions, were referenced to “random motion” baseline 
measures with corresponding gain factors. These baselines were necessary because the visible sur- 
round exhibited a slight random jitter due to system noise even when attempting to track a fixed arti- 
ficial head. 

There are two noteworthy features shown in Figures 4 and 5. First, the gain profiles are clearly 
not flat but instead rise or fall in a frequency-dependent manner. Thus, we must conclude that mov- 
ing the visual stimulus by tracking the subject’s head differs qualitatively from the various other 
forms of altered visual feedback depicted in Figure 3, because those manipulations have yielded 
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Figure 4. Lateral sway gain as a function of frequency when the azimuth of the surrounds tracked the 
subject’s head movements. The four curves represent different gain factor. 



Figure 5. Yaw movement gain as a function of frequency, analogous to Figure 4. 

results that are highly similar across frequency. Second, whether each curve in Figures 4 and 5 rises 
or falls in each frequency band is highly dependent on the sign of the gain factor parameter. For both 
lateral sway (Fig. 4) and yaw movements (Fig. 5), a gain peak associated with positive parameter 
values becomes a gain trough when associated with negative parameter values, and vice versa. 

Even though subjects in the head tracking experiment showed greater stability than did subjects 
who simply closed their eyes, 10 out of the 33 reported symptoms of motion sickness. For 4 of those 

subjects, symptoms persisted more than an hour, and for one subject, persisted 5 hours. Another 
8 subjects found the participation unpleasant but not nauseogenic. In contrast, 3 subjects reported 
extreme exhilaration apparently akin to taking an amusement park ride. Only one subject guessed 
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that the stimulus tracked her head movements, while another subject guessed that her head had been 
captured by motion of the visual stimulus. 


CONCLUSIONS 


We find that exposure to a visual virtual environment enslaved to simulate a subject’s own eye 
movements or head movements can have deleterious effects on the subject’s spatial stability. When 
the entire visible surroundings were moved with the same parameters of angular velocity and motion 
duration which characterize natural saccadic eye movements, subjects tended to lose lateral stability 
(Fig. 1) and to make head yaw movements (Fig. 2, untriggered motions). This pattern of response 
was cancelled, however, when saccade-like stimulus motions were made to coincide with the 
subject’s own voluntary eye movements (also in Fig. 2, triggered by saccade). It seems plausible to 
interpret these findings in the light if theoretical frameworks which propose that a subject generates a 
hypothetical internal signal regarding their own voluntary eye movements. Such an internal signal 
(corollary discharge or efference copy) would presumably serve in the natural world to mitigate the 
effects of retinal image motion accompanying one’s voluntary eye movement. In the present virtual 
world, the same hypothetical internal signal presumably mitigates the effects of external stimulus 
motion if it is made at the same time when this internal signal is active. 

The effects on a subject’s spatial stability when the visual surroundings tracked head movements 
are less readily explained. First, any subject intentions to move the head were discouraged, inasmuch 
as (a) they were instructed to stand still and (b) they became significantly at risk of falling if they 
moved the head voluntarily. It must be questioned whether an efference copy or corollary discharge 
internal signal could exist without subjects’ internal plans to make voluntary movements. Second, 
subjects’ movement gain profiles in Figures 4 and 5 are reminiscent of the changes in poles and 
zeros in the complex domain when a device receiving feedback undergoes a change in its feedback’s 
sign. It is tempting to speculate that head tracking conditions effectively reveal some of the subjects’ 
own internal feedback mechanisms. Unfortunately, the lag time required to update stimulus azimuth 
after sensing head position and orientation (150 msec) was sufficient to complicate detailed interpre- 
tations along these lines. Third, it may be important to understand why so many subjects in the head 
tracking experiments experienced motion sickness symptoms or other discomforts, even though they 
did not become very greatly disoriented. We conjecture that the head-tracking visual feedback must 
have been in conflict with the subjects’ own internal models of their orientation in space. Such 
models are probably multisensory, though highly dependent on visual feedback, and probably estab- 
lished below the level of one’s conscious awareness. 

We believe that the design of virtual environments which are enslaved to the eye or head move- 
ments of a teleoperator must take into account the effects on the human. In the present experiments, 
we found that these effects can include loss of stability, motion sickness, or other undesirable influ- 
ences on situational awareness. These presumably could be mitigated by carefully matching sensor 
resolutions and real time controls of the virtual environment to the capacities of the human 
participants. 
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